Sientia DataOps Model Manager

An enterprise-grade ML model training orchestration platform built on Temporal. Provides robust, scalable workflows for training machine learning models with comprehensive validation, experiment tracking, and automated resource management. Integrates seamlessly with MLFlow for model persistence and PostgreSQL for experiment tracking.

📑 Table of Contents

• Features
  • Core Functionality
  • Advanced Capabilities
  • Development & Quality Assurance
• Architecture
  • Architecture Principles
  • Key Components
  • Data Flow Architecture
  • Security Architecture
• Workflows
  • Train Model Workflow
  • Cleanup Files Workflow
• Installation & Setup
  • Prerequisites
  • Environment Setup
  • Temporal Namespace Setup
  • Local Development Setup
    • Port Forward Setup Script
• How to Run
  • Running the Model Manager Application
  • Running Tests and Coverage
  • Manual Test Execution
  • Manual Application Execution
• Code Quality & Validation
  • Overview
  • Validation Tools
  • Tools Installation
  • Complete Validation
  • Automatic Fixes
  • Configuration
  • CI/CD Integration
  • Best Practices
• Testing
  • Test Structure
  • Test Execution
  • Integration Tests
    • Running Integration Tests
    • Test Scenarios
    • Scenario File Structure
    • Creating New Scenarios
    • Important Validations
• Monitoring and Metrics
  • Application Health Metrics
  • Training Metrics
• Configuration
  • Environment Variables
  • Workflow Configuration
• Development
  • Code Quality & Testing
  • Project Structure
  • Adding New Features
  • Test Coverage Guidelines
• Troubleshooting
  • Common Issues
  • Debug Mode
• Performance Tuning
  • Key Parameters
  • Scaling Considerations
• Contributing
  • Code Quality Standards
• License
• Support
• Local GitHub Actions Testing (act)
  • What is act?
  • Installation
  • Configuration
  • Usage
  • Command Reference
• Docker
• Helm Chart

Features

Core Functionality

• ML Model Training Pipeline: Complete training workflow from validation to deployment using MLFlow
• Polynomial Regression Support: Configurable polynomial degree with interaction terms and mandatory scaler validation
• Automated File Cleanup: Scheduled cleanup of stale files from MinIO and local filesystem
• Temporal Workflow Orchestration: Robust workflow management with granular retry policies and fault tolerance
• Parameter Validation: Defense-in-depth validation with business rules and type checking
• Experiment Tracking: Comprehensive status tracking in PostgreSQL database
• Resource Management: Automatic cleanup of temporary files and storage
• Comprehensive Monitoring: Prometheus metrics and detailed logging for operational visibility
• Scheduled Jobs: Automated daily cleanup with configurable cron schedules

Advanced Capabilities

• Granular Retry Policies: Different strategies for network, training, MLFlow, database, and filesystem operations
• Configurable Timeouts: Environment variable-based timeouts supporting large training files (up to 200MB)
• Notification System: Integrated alerting and notification management via MongoDB
• Scalable Architecture: Kubernetes-ready deployment with horizontal scaling support
• MLFlow Integration: Seamless model and artifact persistence to MLFlow tracking server
• Per-Variable Lag Configuration: Flexible lag settings for each variable independently
• Date Range Filtering: Filter training data by start/end dates and removed intervals
• NaN Treatment Options: Configurable handling of missing values (drop, linear interpolation)
• RCE Drift Metrics: Reduced Coulomb Energy metrics for drift detection

Development & Quality Assurance

• Code Quality Tools: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
• Automated Validation: Pre-commit validation script (validate.sh) and CI/CD integration
• Comprehensive Testing: pytest with async support and 100% code coverage 🎯
• Type Safety: Static type checking with mypy for improved code reliability
• Coverage Visualization: Integration with Coverage Gutters for real-time coverage feedback
• Automated Versioning: Semantic versioning based on branch patterns (release/, feature/, fix/, rc/)
• Integration Test Scenarios: JSON-based test scenarios with batch execution support

Architecture

The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.

Architecture Principles

1. Separation of Concerns

• Worker Layer: Manages Temporal workers, task queues, and application lifecycle
• Workflow Layer: Orchestrates business logic and process coordination
• Activity Layer: Implements specific operations and external system interactions
• Data Layer: Handles data persistence, caching, and external service connections

2. Fault Tolerance & Resilience

• Automatic Retry Policies: Configurable retry strategies for transient failures
• Circuit Breaker Pattern: Prevents cascading failures in external service calls
• Graceful Degradation: System continues operating with reduced functionality
• Comprehensive Error Handling: Detailed error reporting and notification integration

3. Scalability & Performance

• Horizontal Scaling: Multiple worker instances for load distribution
• Task Queue Isolation: Separate queues for different workflow types
• Connection Pooling: Optimized database and external service connections
• Asynchronous Processing: Non-blocking operations for improved throughput

4. Observability & Monitoring

• Prometheus Metrics: Comprehensive system and business metrics
• Structured Logging: Consistent log format with correlation IDs
• Health Checks: Endpoint health monitoring and alerting
• Performance Tracing: Request flow tracking and bottleneck identification

Key Components

Worker (model_manager/worker/worker.py)

• Purpose: Main application orchestrator managing Temporal workers and task queues
• Responsibilities:
  • Temporal client initialization and connection management
  • Worker lifecycle management and graceful shutdown
  • Task queue configuration and load balancing
  • Prometheus metrics server initialization
  • Notification handler setup and configuration
• Key Features:
  • Automatic scaling with PollerBehaviorAutoscaling
  • Health check endpoints for Kubernetes liveness/readiness probes
  • Graceful shutdown with cleanup procedures
  • Multi-instance deployment support
  • Two dedicated task queues:
    • train_model-queue: ML model training workflows
    • cleanup-queue: File cleanup workflows
  • Automated cleanup schedule management

Workflows (model_manager/workflows/)

• TrainModel: Complete ML model training pipeline from validation to deployment
• CleanupFiles: Automated cleanup of stale files from MinIO and local filesystem
• Key Features:
  • Temporal workflow definitions with granular retry policies
  • Parameter validation with business rules
  • Comprehensive error handling and status tracking
  • Configurable timeouts for different operation types
  • Automatic resource cleanup and management
  • Scheduled cleanup jobs with cron expressions

Activities (model_manager/activities/)

• Activities: Main activity orchestrator combining all functionality through multiple inheritance
• ExperimentTracking: ML experiment lifecycle tracking and database operations
  • Unified update_experiment_run() method for all experiment status updates
  • Support for three update types: STATUS, STATUS_WITH_ERROR, MODEL_SAVED
  • Automatic error message truncation (1024 chars)
  • Connection pooling and retry logic
• Training: ML model training operations with MLFlow and MinIO integration
  • Unified train_model() method for complete training pipeline
  • Receives pre-downloaded files (BytesIO) to avoid memory leaks
  • Returns success/failure status with TrainModelResult or error message
  • No exception raising on failure - allows workflow to handle errors gracefully
  • Integration with TrainingRepository for business logic separation
  • MLFlow model saving and artifact management
  • MinIO object storage operations
  • Polynomial Regression: Support for configurable degree and interaction terms
  • Training Predictions: Calculates y_train_pred before denormalization for accurate metrics
• Cleanup: File and directory cleanup operations
  • cleanup_minio_files(): Removes stale files from MinIO based on timestamp prefixes
  • cleanup_temp_directories(): Cleans local temporary directories
  • Configurable retention period (default: 24 hours)
  • Dry-run mode for testing
• Key Features:
  • Multiple inheritance pattern for unified activity interface
  • Parameter validation with business rules
  • MLFlow integration for model persistence
  • Comprehensive error handling and notification integration
  • Experiment tracking with automatic status management
  • Timestamp-based file cleanup with regex pattern matching

Data Services (model_manager/utils/)

• Connectors Config: Environment variable-based configuration management
• Repository: Data access layer for training and MLFlow operations
  • training_repository.py: Training business logic and operations
  • model_repository.py: MLFlow artifact generation and model persistence
• Models: Data models and schemas
  • train_model_params.py: Training parameters model with comprehensive validation
  • train_model_result.py: Training result model (includes y_train_pred)
  • experiment_status.py: Experiment status enum
• Key Features:
  • Environment variable-based configuration with sensible defaults
  • Connection pool management and optimization
  • Security credential management
  • Configuration validation and error handling
  • Type-safe data models with validation

Data Flow Architecture

Model Training Pipeline

Training Request → Parameter Validation → MinIO Data Download → 
Model Training → MLFlow Model Save → Resource Cleanup → Status Update

Key Stages:

1. Validation: Experiment run ID and training parameters validation
2. Data Acquisition: Download training data from MinIO storage
3. Training: Execute ML model training with validated parameters
4. Persistence: Save trained model and artifacts to MLFlow
5. Cleanup: Remove temporary files and update experiment status

Security Architecture

Authentication & Authorization

• MLFlow API Authentication: Username/password with secure transmission
• Database Connection Security: Encrypted connections with credential management
• Kubernetes Secrets Integration: Secure credential storage and access

Network Security

• TLS/SSL Encryption: Secure communication channels
• Network Isolation: Kubernetes network policies and service mesh
• Firewall Rules: Controlled access to external services
• VPN Integration: Secure remote access and management

Data Security

• Data Encryption: At-rest and in-transit encryption
• Access Control: Role-based access control (RBAC)
• Audit Logging: Comprehensive access and operation logging
• Data Retention: Configurable data lifecycle management

Workflows

Train Model Workflow (train_model.py)

The TrainModel workflow orchestrates the complete ML model training pipeline from parameter validation through model saving and cleanup.

Purpose

• Model Training: Complete ML model training pipeline
• Parameter Validation: Defense-in-depth validation with business rules
• Resource Management: Automatic cleanup of temporary resources
• Status Tracking: Comprehensive experiment tracking in database
• Error Handling: Robust error handling with detailed context logging

Execution Flow

1. Validate Experiment Run ID: Critical validation before any DB updates
2. Validate Training Parameters: Type checking + business rules validation
3. Download Training Data: Fetch file from MinIO storage
4. Train Model: Execute ML model training with validated parameters
5. Save to MLFlow: Save trained model and artifacts to MLFlow
6. Cleanup Resources: Delete temporary files and MinIO data

Key Features

• Granular Retry Policies: Different strategies for network, training, MLFlow, database, and filesystem operations
• Configurable Timeouts: Environment variable-based timeouts supporting files up to 200MB
• Idempotent Cleanup: Safe replay with Temporal workflow replay mechanism
• Structured Logging: Rich context in error messages for debugging
• Business Validation: 10 business rules including range checks, consistency validation, and data integrity

Input Parameters

{
  "experiment_run_id": 123,
  "target_variable": "price",
  "variable_columns": ["feature1", "feature2"],
  "train_size": 80,
  "shuffle": true,
  "use_scaler": true,
  "include_ar": false,
  "bucket_name": "ml-data",
  "file_name": "training_data.csv",
  "line_separator": "\n",
  "decimal_separator": ".",
  "lag_train": {"feature1": 0, "feature2": 2},
  "lag_val": {"feature1": 0, "feature2": 1},
  "rem_static_win": false,
  "low_lim": {"feature1": 0.0, "feature2": 0.0},
  "upp_lim": {"feature1": 100.0, "feature2": 100.0},
  "window": 10,
  "experiment_name": "production_model_v1",
  "removed_intervals": [],
  "model_name": "Linear Regression",
  "degree": 1,
  "interaction_only": false,
  "nan_treatment": "drop",
  "start_date": null,
  "end_date": null,
  "scaler_name": "Standard Scaler",
  "support_filters": {}
}

Architecture Diagram

flowchart TD
    A[1. validate_experiment_run_id] --> B[2. validate_train_params]
    B --> C[3. fetch_file_from_minio]
    C --> D[4. train_model]
    D --> E[5. save_model]
    E --> F[6. cleanup_run_directory]
    F --> G[7. delete_file_from_minio]
    
    B -.-> DB[(PostgreSQL)]
    C -.-> MinIO[MinIO Storage]
    D -.-> Training[ML Training]
    E -.-> MLFlow[MLFlow]
    F -.-> FS[Filesystem]
    G -.-> MinIO

Retry Strategies

The workflow implements 5 different retry policies optimized for each operation type:

Operation Type	Initial Interval	Max Interval	Backoff	Max Attempts	Use Case
Network	1s	10s	2.0x	5	MinIO operations (transient network errors)
No Retry	-	-	-	1	Training/Validation (permanent data errors)
MLFlow	5s	30s	2.0x	3	MLFlow operations (API timeouts)
Database	2s	20s	2.0x	5	PostgreSQL updates (lock contention)
Filesystem	2s	10s	1.5x	3	Cleanup operations (busy resources)

Business Validation Rules

The workflow validates comprehensive business rules beyond type checking:

1. train_size: Must be between 10-100%
2. variable_columns: Cannot be empty
3. lag_train, lag_val: Per-variable dictionaries with non-negative values
4. window: Must be non-negative integer
5. low_lim/upp_lim: Must have same keys and low < upp for each variable
6. target_variable: Cannot be empty
7. bucket_name, file_name, experiment_name: Cannot be empty or whitespace
8. degree: Must be at least 1; must be >= 2 for Polynomial Regression
9. nan_treatment: Must be one of 'drop', 'linear interpolation', 'fill linear'
10. scaler_name: Must be 'Standard Scaler' or 'None'
11. model_name: Must be 'Linear Regression' or 'Polynomial Regression'
12. Polynomial Regression requires Scaler: Models with degree > 1 must have a scaler to prevent numerical overflow
13. Linear Regression requires degree 1: Linear models must have degree = 1

Cleanup Files Workflow (cleanup_files.py)

The CleanupFiles workflow provides automated cleanup of stale files from MinIO storage and local temporary directories. It runs on a scheduled basis (default: daily at midnight UTC) to maintain storage hygiene.

Purpose

• Storage Management: Automatic removal of old files from MinIO and local filesystem
• Retention Policy: Configurable retention period (default: 24 hours)
• Scheduled Execution: Cron-based scheduling for automated cleanup
• Resource Optimization: Prevents storage bloat and reduces costs

Execution Flow

1. Cleanup MinIO Files: Scan and delete files older than retention period from MinIO bucket
2. Cleanup Local Directories: Remove temporary directories older than retention period

Key Features

• Timestamp-Based Cleanup: Uses filename/directory timestamps for age determination
• Pattern Matching: Regex patterns for MinIO (timestamp-filename) and directories (name_YYYYMMDD_HHMMSS_microseconds)
• Configurable Retention: Environment variable-based retention period
• Dry-Run Mode: Test cleanup operations without actual deletion
• Idempotent: Safe to run multiple times
• Error Handling: Continues cleanup even if individual operations fail

Input Parameters

{
  "bucket_name": "model-training"  // Optional, defaults to DEFAULT_CLEANUP_BUCKET env var
}

Schedule Configuration

The cleanup schedule is automatically created when the worker starts:

Configuration	Environment Variable	Default	Description
Schedule ID	CLEANUP_SCHEDULE_ID	cleanup-files-daily	Unique identifier for the schedule
Cron Expression	CLEANUP_CRON	0 0 * * *	Daily at midnight UTC
Timezone	CLEANUP_TIMEZONE	UTC	Timezone for cron execution
Task Queue	CLEANUP_TASK_QUEUE	cleanup-queue	Dedicated task queue
Execution Timeout	CLEANUP_EXECUTION_TIMEOUT_HOURS	1	Maximum execution time (hours)
Retention Period	CLEANUP_RETENTION_HOURS	24	Files older than this are deleted
Dry Run	CLEANUP_DRY_RUN	false	Test mode without actual deletion
Max Keys	MAX_KEYS_CLEANUP	1000	MinIO list operation page size

Architecture Diagram

flowchart TD
    A[Scheduled Trigger] --> B[cleanup_minio_files]
    B --> C[cleanup_temp_directories]
    
    B -.-> MinIO[MinIO Storage]
    C -.-> FS[Local Filesystem]

Retry Strategies

Operation Type	Initial Interval	Max Interval	Backoff	Max Attempts	Use Case
Network	1s	10s	2.0x	5	MinIO operations (transient network errors)
No Retry	-	-	-	1	Local filesystem operations (permanent errors)

Cleanup Patterns

MinIO Files:

• Pattern: {timestamp}-{filename} where timestamp is milliseconds since epoch
• Example: 1638360000000-training_data.csv
• Retention: Files older than CLEANUP_RETENTION_HOURS are deleted

Local Directories:

• Pattern: {name}_{YYYYMMDD}_{HHMMSS}_{microseconds}
• Example: temp_20231201_143052_123456
• Retention: Directories older than CLEANUP_RETENTION_HOURS are deleted

Installation & Setup

Prerequisites

• Python 3.11+
• Temporal server/cluster
• PostgreSQL database
• MLFlow server
• MinIO object storage (for MLFlow artifacts)
• MongoDB server (for notifications)

Note: External dependencies must be available either through:

• Kubernetes cluster deployment
• Docker Compose setup
• Cloud-managed services
• Local installations

MinIO Setup

MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:

📚 Install MinIO via Helm Chart on K8s

This guide covers:

• Helm chart installation on Kubernetes
• Storage configuration and persistence
• Access credentials setup
• Integration with MLFlow

Environment Setup

1. Clone the repository:

   git clone <repository-url>
   cd sientia-dataops-model-manager
   

2. Create virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   

3. Install dependencies:

   pip install -r requirements.txt
   

4. Configure environment variables (see Configuration section)

5. Run validation script:

   ./validate.sh
   

Temporal Namespace Setup

The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created before starting the application.

Why Create a Namespace?

• Isolation: Separates Model Manager workflows from other applications
• Retention Control: Configures workflow history retention (default: 7 days)
• Multi-tenancy: Enables multiple environments (dev, staging, prod) on same cluster
• Security: Allows namespace-level access control and permissions

When to Create?

• ✅ Before first deployment in any environment
• ✅ Once per environment (dev, staging, production)
• ✅ After Temporal cluster setup or upgrade

How to Create the Namespace

Option 1: Using Temporal Admin Tools Pod (Recommended for Kubernetes)

# 1. List Temporal pods
kubectl get pods -n temporal

# 2. Connect to admin tools pod
kubectl exec -it -n temporal <temporal-admin-tools-pod-name> -- bash

# 3. Create namespace
tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

# 4. Verify creation
tctl --namespace model-manager namespace describe

# 5. Exit pod
exit

Option 2: Using Port Forward (Local Development)

# 1. Port forward Temporal frontend
kubectl port-forward -n temporal svc/temporal-frontend 7233:7233

# 2. In another terminal, create namespace
tctl --address localhost:7233 \
     --namespace model-manager \
     namespace register \
     --retention 7 \
     --description "Model Manager - ML Model Orchestration Namespace"

# 3. Verify
tctl --address localhost:7233 --namespace model-manager namespace describe

Option 3: Direct kubectl exec (One-liner)

kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace register \
    --retention 7 \
    --description "Model Manager - ML Model Orchestration Namespace"

Namespace Configuration

Parameter	Value	Description
Name	model-manager	Namespace identifier (configurable via TEMPORAL_NAMESPACE env var)
Retention	7 days	Workflow history retention period
Description	Model Manager - ML Model Orchestration Namespace	Human-readable description

Verification

To verify the namespace was created successfully:

# List all namespaces
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- tctl namespace list

# Describe specific namespace
kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    tctl --namespace model-manager namespace describe

Troubleshooting

Error: "namespace already exists"

• ✅ This is fine! The namespace is already configured
• No action needed, proceed with application deployment

Error: "connection refused"

• ❌ Temporal server is not accessible
• Verify Temporal cluster is running: kubectl get pods -n temporal
• Check network connectivity and port forwarding

Error: "permission denied"

• ❌ Insufficient permissions to create namespace
• Contact cluster administrator for namespace creation
• Or request elevated permissions for your service account

Local Development Setup

1. Clone the repository

   git clone <repository-url>
   cd sientia-dataops-model-manager
   

2. Create virtual environment

   conda create -p ./venv python=3.11
   conda activate ./venv
   

3. Install dependencies

4. Install github cli bash sudo apt update sudo apt install gh -y

5. Authenticate with github bash gh auth login

6. Install Python dependencies bash python -m pip install --upgrade pip # Install production dependencies pip install -r requirements.txt # Install development and testing tools pip install -r requirements-dev.txt

7. Create environment configuration file

   cp .env.example .env
   # Edit .env with your connection details
   

8. Configure external dependencies

   The Model Manager requires connections to several external services. For local development, you can use the provided port-forward script to establish connections to services running in your Kubernetes cluster.

Services and Ports:

Local Port	Service	Description	Namespace
5432	paradedb-rw	PostgreSQL Database	paradedb
45249	sientia-tracker-mlflow-tracking	MLflow Tracking Server	sientia-tracker
37463	temporal-frontend	Temporal gRPC API	temporal
8080	temporal-web	Temporal Web UI	temporal
42297	my-release-mongodb	MongoDB Database	mongodb
36577	minio	MinIO Object Storage	minio

Managing Port Forwards:

# View active port forwards
jobs -l

# Stop all port forwards
jobs -p | xargs kill

Manual Port Forwarding:

If you prefer manual control or need different ports:

# PostgreSQL
kubectl -n paradedb port-forward svc/paradedb-rw 5432:5432 &

# MLflow
kubectl -n sientia-tracker port-forward svc/sientia-tracker-mlflow-tracking 45249:80 &

# Temporal
kubectl -n temporal port-forward svc/temporal-frontend 37463:7233 &

# Temporal UI
kubectl -n temporal port-forward svc/temporal-web 8080:8080 &

# MongoDB
kubectl -n mongodb port-forward svc/my-release-mongodb 42297:27017 &

# MinIO
kubectl -n minio port-forward svc/minio 36577:9000 &

How to Run

Running the Model Manager Application

Use the provided script to run the application locally:

# Make script executable (first time only)
chmod +x run_local.sh

# Run the application
./run_local.sh

The script will:

• Load environment variables from .env
• Start the model-manager worker application

Note: Activate your virtual environment before running the script:

source ./venv/bin/activate  # or: conda activate ./venv
./run_local.sh

Running Tests and Coverage

Run tests with coverage using pytest directly:

# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=html

# Open the coverage report in your browser
open htmlcov/index.html  # macOS
xdg-open htmlcov/index.html  # Linux

Manual Test Execution

You can also run tests manually:

# Activate virtual environment
source ./venv/bin/activate

# Run all tests
pytest

# Run with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test categories
pytest tests/activities/
pytest tests/workflow/

Manual Application Execution

For manual execution without scripts:

# Activate virtual environment
source ./venv/bin/activate

# Load environment variables (if using .env file)
if [ -f .env ]; then
    export $(cat .env | grep -v '^#' | xargs)
fi

# Start the model-manager worker
python -m model_manager.worker.worker

Code Quality & Validation

Overview

Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.

Validation Tools

1. Ruff - Linting and Formatting ⚡

Modern and extremely fast tool (written in Rust) that replaces multiple tools:

• Linting: Detects code errors, style issues (PEP 8), common bugs
• Formatting: Automatically formats code consistently
• Speed: 10-100x faster than Flake8/Black

2. mypy - Type Checking 🏷️

Static type checker that analyzes type hints:

• Detects type errors before execution
• Improves code documentation
• Prevents bugs related to incorrect types

3. Bandit - Security Analysis 🔒

Security vulnerability scanner:

• Detects insecure code patterns
• Identifies hardcoded passwords, SQL injection, etc.
• Ensures compliance with security practices

4. pytest - Automated Testing 🧪

Testing framework with code coverage:

• Executes unit and integration tests
• Measures code coverage
• Supports asynchronous tests

Tools Installation

# Install development dependencies
pip install -r requirements-dev.txt

Complete Validation

Option 1: Automated Script (Recommended)

# Run all validations at once
./validate.sh

The validate.sh script automatically executes:

1. ✅ Format checking (Ruff)
2. ✅ Code linting (Ruff)
3. ✅ Type checking (mypy)
4. ✅ Security analysis (Bandit)
5. ✅ Unit tests with coverage (pytest)

Option 2: Individual Commands

# 1. Check formatting
ruff format --check model_manager/ tests/

# 2. Check linting
ruff check model_manager/ tests/

# 3. Check types
mypy model_manager/

# 4. Security analysis
bandit -r model_manager/ -ll

# 5. Run tests
pytest tests/ --cov=model_manager --cov-report=term-missing

Automatic Fixes

Some tools can automatically fix issues:

# Format code automatically
ruff format model_manager/ tests/

# Fix linting issues automatically
ruff check --fix model_manager/ tests/

Configuration

All tools are configured in the pyproject.toml file:

• Ruff: Linting rules, formatting, complexity
• mypy: Type checking settings
• pytest: Test and coverage options
• Bandit: Security rules

CI/CD Integration

The project uses reusable workflows from Aignosi/github_workflow_templates for CI/CD:

Quality Gate (.github/workflows/quality-gate.yml)

Runs automatically on each Pull Request to main:

• ✅ Formatting and linting block merge if they fail
• ⚠️ Type checking and security generate warnings but don't block
• ✅ Tests must pass with minimum 80% coverage
• ✅ SonarQube analysis for code quality metrics
• ✅ Automatic version calculation based on branch pattern

Deploy (.github/workflows/deploy.yml)

Runs automatically when a PR is merged to main:

• ✅ Builds and pushes Docker image to Azure Container Registry
• ✅ Creates GitHub release with calculated version
• ✅ Deploys to Kubernetes using Helm

Best Practices

1. Before Commit: Run ./validate.sh to ensure quality
2. During Development: Use ruff check --watch for real-time feedback
3. Type Hints: Add type hints to new functions for better validation
4. Tests: Maintain coverage above 80%
5. Security: Review and fix all Bandit warnings

Testing

Test Structure

tests/
├── activities/      # Activity implementation tests
├── workflows/       # Workflow orchestration tests
├── utils/           # Utility function tests
├── worker/          # Worker tests
├── schedules/       # Schedule configuration tests
└── sientia/         # Sientia module tests

Test Execution

# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run tests with coverage
pytest --cov=model_manager --cov-report=html

# Run specific test modules
pytest tests/activities/test_training.py
pytest tests/workflows/test_train_model.py

Integration Tests

The project includes integration tests that validate the complete training workflow against a running Temporal cluster. These tests use JSON-based scenario files for easy configuration and maintenance.

Running Integration Tests

# List available scenarios
python scripts/run_training_test.py --list

# Run a specific test scenario
python scripts/run_training_test.py --scenario 01-linear-regression-basic

# Run with custom CSV data file
python scripts/run_training_test.py --scenario 03-polynomial-regression-degree2 --csv /path/to/data.csv

# Run ALL scenarios sequentially with summary report
python scripts/run_training_test.py --all

# Run all scenarios with custom CSV
python scripts/run_training_test.py --all --csv docs/custom-data.csv

Batch Execution Output

When running all scenarios with --all, the script provides:

• Progress indicators for each scenario ([1/10] Running scenario: ...)
• Status symbols (✓ for passed, ✗ for failed)
• Final summary with total/passed/failed counts
• Detailed error messages for failed scenarios
• Exit code 0 if all pass, 1 if any fail

Example output:

Running 10 scenarios...

[1/10] Running scenario: 01-linear-regression-basic
  Loaded scenario: 01-linear-regression-basic
  Uploaded CSV to MinIO: test-model-data-20231219-120000.csv
  Created experiment_run with ID: 42
  Workflow started: train-model-test-abc123
[1/10] ✓ 01-linear-regression-basic

...

============================================================
SUMMARY
============================================================
Total: 10 | Passed: 9 | Failed: 1
============================================================

✓ PASSED:
  - 01-linear-regression-basic
  - 02-linear-regression-with-scaler
  ...

✗ FAILED:
  - 05-linear-regression-with-lags
    Error: Failed to start Temporal workflow: connection refused

Test Scenarios

Test scenarios are defined as JSON files in docs/test-scenarios/. Each scenario configures a complete training workflow with specific parameters:

Scenario	Description	Key Features
01-linear-regression-basic	Basic linear regression	No scaler, no lags
02-linear-regression-with-scaler	Linear regression with normalization	Standard Scaler enabled
03-polynomial-regression-degree2	Polynomial regression (degree 2)	Requires scaler (mandatory)
04-polynomial-regression-degree3	Polynomial regression (degree 3)	Requires scaler (mandatory)
05-linear-regression-with-lags	Linear regression with lag features	Lag train/val configuration
06-linear-regression-nan-interpolation	Linear regression with NaN handling	nanTreatment: "linear interpolation"
07-linear-regression-static-window-removal	Linear regression with static window removal	remStaticWin: true
08-linear-regression-with-limits	Linear regression with variable limits	lowLim/uppLim configuration
09-polynomial-degree2-with-scaler-and-lags	Complete polynomial scenario	Scaler + lags + degree 2
10-linear-regression-with-ar	Linear regression with autoregressive variable	includeAr: true

Scenario File Structure

{
  "_description": "Human-readable description of the scenario",
  "experimentName": "test-experiment-name",
  "username": "user@example.com",
  "modelName": "Linear Regression",
  "targetVariable": "target_column_name",
  "variableColumns": ["feature1", "feature2"],
  "lagTrain": {"feature1": 0, "feature2": 0},
  "lagVal": {"feature1": 0, "feature2": 0},
  "remStaticWin": false,
  "lowLim": {},
  "uppLim": {},
  "window": 0,
  "useScaler": false,
  "includeAr": false,
  "trainSize": 80,
  "shuffle": true,
  "lineSeparator": ",",
  "decimalSeparator": ".",
  "removedIntervals": [],
  "degree": 1,
  "interactionOnly": false,
  "nanTreatment": "drop",
  "startDate": null,
  "endDate": null,
  "scalerName": "None",
  "supportFilters": {}
}

Creating New Scenarios

1. Copy an existing scenario file as a template
2. Modify parameters according to your test case
3. Save with a descriptive name: XX-description.json
4. Run with: python scripts/run_training_test.py --scenario XX-description

Important Validations

The training workflow enforces several business rules:

• Polynomial Regression requires Scaler: Models with degree > 1 must have useScaler: true and a valid scalerName to prevent numerical overflow
• Static Window Removal requires DatetimeIndex: Scenarios with remStaticWin: true require data with a timestamp column for the TimeSeriesDiscontinuityAnalyzer
• Variable Limits Consistency: lowLim and uppLim must have matching keys, and lowLim[key] < uppLim[key] for all variables

Monitoring and Metrics

The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:

Application Health Metrics

• app_up: Application health status (1=healthy, 0=unhealthy)
  • Labels: pod_id

Workflow Execution Metrics

• workflow_execution_total: Total workflow executions
  • Labels: workflow_name, status (success/failure)
• activity_execution_total: Total activity executions
  • Labels: activity_name, status (success/failure)

Training Metrics

• Training success/failure rates through notification system
• Model save performance metrics
• Experiment status tracking
• RCE Drift Metrics: Reduced Coulomb Energy (RCE) for drift detection
  • silverman_radius: Optimal bandwidth for kernel density estimation
  • rce_reference: RCE value for reference data
  • rce_current: RCE value for current data
  • rce_drift: Drift score between reference and current distributions

Cleanup Metrics

• Cleanup execution success/failure rates
• Number of files deleted from MinIO
• Number of directories cleaned from local filesystem
• Cleanup duration and performance

Configuration

Environment Variables

Variable	Description	Default	Required
TEMPORAL_HOST	Temporal server address	localhost:7233	Yes
TEMPORAL_NAMESPACE	Temporal namespace	model-manager	No
TEMPORAL_USE_TLS	Enable TLS for Temporal connection	false	No
TRAIN_TASK_QUEUE	Task queue for training workflows	train_model-queue	No
CLEANUP_TASK_QUEUE	Task queue for cleanup workflows	cleanup-queue	No
POSTGRES_HOST	PostgreSQL hostname	localhost	Yes
POSTGRES_PORT	PostgreSQL port	5432	Yes
POSTGRES_USER	PostgreSQL username	sientia	Yes
POSTGRES_PASSWORD	PostgreSQL password	sientia	Yes
POSTGRES_DBNAME	PostgreSQL database	sientia	Yes
POSTGRES_MIN_CONNECTIONS	Minimum PostgreSQL connections	5	No
POSTGRES_MAX_CONNECTIONS	Maximum PostgreSQL connections	20	No
MLFLOW_URL	MLFlow server URL (full URL with protocol and port)	http://localhost:5080	Yes
MLFLOW_USERNAME	MLFlow username	aignosi	Yes
MLFLOW_PASSWORD	MLFlow password	aignosi	Yes
MINIO_ENDPOINT_URL	MinIO server endpoint	http://minio.minio.svc.cluster.local:9000	Yes
MINIO_ACCESS_KEY	MinIO access key	minioadmin	Yes
MINIO_SECRET_KEY	MinIO secret key	minioadmin	Yes
MINIO_REGION	MinIO region	us-east-1	No
MINIO_USE_SSL	Enable SSL for MinIO	false	No
MINIO_MAX_RETRY_ATTEMPTS	Maximum retry attempts	3	No
MINIO_RETRY_MODE	Retry mode (standard/adaptive)	adaptive	No
MINIO_CONNECT_TIMEOUT	Connection timeout (seconds)	10	No
MINIO_READ_TIMEOUT	Read timeout (seconds)	60	No
MONGODB_URL	MongoDB connection URI	localhost:27018	Yes
MONGODB_USERNAME	MongoDB username	root	Yes
MONGODB_PASSWORD	MongoDB password	wKZDbMNU1c	Yes
MONGODB_DATABASE	MongoDB database name	sientia	Yes
MONGODB_TTL_INDEX_HOURS	MongoDB TTL index hours	1	No
CLEANUP_SCHEDULE_ID	Cleanup schedule identifier	cleanup-files-daily	No
CLEANUP_CRON	Cleanup cron expression	0 0 * * *	No
CLEANUP_TIMEZONE	Cleanup schedule timezone	UTC	No
CLEANUP_EXECUTION_TIMEOUT_HOURS	Cleanup execution timeout	1	No
CLEANUP_RETENTION_HOURS	File retention period (hours)	24	No
CLEANUP_DRY_RUN	Dry-run mode (no actual deletion)	false	No
MAX_KEYS_CLEANUP	MinIO list operation page size	1000	No
DEFAULT_CLEANUP_BUCKET	Default bucket for cleanup	model-training	No
LOG_LEVEL	Application log level	INFO	No
PROJECT_NAME	Project name for metrics	model-manager	No
HTTP_METRICS_PORT	Prometheus metrics port	9090	No
HTTP_SDK_METRICS_PORT	Temporal SDK metrics port	9091	No
POD_ID	Kubernetes pod identifier	None	No
EXTRA_PIP_REQUIREMENTS	Extra pip requirements for MLFlow model serving	None	No

Workflow Activity Timeouts

These timeouts control how long each activity in workflows can run before timing out. All values are in seconds.

Training Workflow Timeouts:

Variable	Description	Default	Calculation Basis
TIMEOUT_VALIDATE_PARAMS	Parameter validation timeout	30	Fast operation, no I/O
TIMEOUT_TRAIN_MODEL	Model training timeout	2700	Large dataset processing (45 min)
TIMEOUT_DELETE_FILE	Delete file from MinIO timeout	120	MinIO delete operation (2 min)
TIMEOUT_UPDATE_DATABASE	Database update timeout	30	PostgreSQL update query (30 sec)

Cleanup Workflow Timeouts:

Variable	Description	Default	Calculation Basis
TIMEOUT_CLEANUP_MINIO	MinIO cleanup timeout	300	Scan and delete multiple files (5 min)
TIMEOUT_CLEANUP_LOCAL	Local cleanup timeout	120	Scan and delete directories (2 min)

Note: These timeouts can be adjusted based on your infrastructure performance and file sizes. If you're processing files larger than 200MB or have slower network/compute resources, increase these values accordingly.

Development

Code Quality & Testing

The project maintains 100% code coverage with comprehensive unit and integration tests.

Running Tests

# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html

# Run specific test file
pytest tests/activities/test_training.py -v

# Run with coverage visualization
pytest tests/ --cov=model_manager --cov-report=xml
# Then open htmlcov/index.html in browser

Validation Script

The validate.sh script runs all quality checks before commit:

./validate.sh

This script performs:

1. ✅ Code Formatting (Ruff)
2. ✅ Code Linting (Ruff)
3. ✅ Type Checking (mypy)
4. ✅ Security Analysis (Bandit)
5. ✅ Unit Tests (pytest with 80%+ coverage requirement)

Coverage Visualization

For real-time coverage feedback in VS Code/Windsurf:

1. Install Coverage Gutters extension
2. Configure .vscode/settings.json:

   {
     "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
     "coverage-gutters.coverageFileNames": ["coverage.xml"],
     "coverage-gutters.showLineCoverage": true,
     "coverage-gutters.showRulerCoverage": true
   }
   

3. Run tests to generate coverage:

   pytest tests/ --cov=model_manager --cov-report=xml
   

4. Activate Coverage Gutters: Press Ctrl+Shift+7 (or Cmd+Shift+7 on Mac)

Automated Versioning

The project uses semantic versioning based on branch patterns:

Branch Pattern	Version Change	Example
release/*	Major version bump	2.0.0
feature/*	Minor version bump	1.2.0
fix/*	Patch version bump	1.1.3
rc/*	Release candidate	1.1.2-rc2

Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.

Project Structure

sientia-dataops-model-manager/
├── model_manager/           # Main application package
│   ├── activities/          # Temporal activity implementations
│   │   ├── __init__.py
│   │   ├── activities.py   # Main activities orchestrator (combines all activities)
│   │   ├── experiment_tracking.py # Experiment status tracking and database operations
│   │   ├── training.py     # ML model training operations (includes MLFlow & MinIO)
│   │   └── cleanup.py      # File and directory cleanup operations
│   ├── workflows/           # Temporal workflow definitions
│   │   ├── __init__.py
│   │   ├── train_model.py  # Complete ML model training workflow
│   │   └── cleanup_files.py # Automated file cleanup workflow
│   ├── schedules/           # Temporal schedule configurations
│   │   ├── __init__.py
│   │   └── cleanup_schedule.py # Cleanup schedule creation and management
│   ├── worker/              # Worker implementation
│   │   ├── __init__.py
│   │   └── worker.py       # Main worker orchestrator (Temporal client setup)
│   ├── utils/               # Utility functions and helpers
│   │   ├── __init__.py
│   │   ├── connectors_config.py # Environment-based configuration builders
│   │   ├── exceptions.py   # Custom exception definitions
│   │   ├── logger_helper.py # Logger initialization utilities
│   │   ├── models/         # Data models and schemas
│   │   │   ├── __init__.py
│   │   │   ├── train_model_params.py # Training parameters model
│   │   │   ├── train_model_result.py # Training result model
│   │   │   └── experiment_status.py  # Experiment status enum
│   │   └── repository/     # Data access layer
│   │       ├── training_repository.py # Training business logic
│   │       ├── model_repository.py    # MLFlow artifact management
│   │       └── storage_repository.py  # MinIO storage operations
│   ├── sientia/             # Sientia-specific implementations
│   │   ├── __init__.py
│   │   ├── exceptions.py   # Custom exceptions
│   │   ├── metrics.py      # Business metrics (includes RCE drift detection)
│   │   ├── models.py       # ML model implementations (Linear & Polynomial Regression)
│   │   ├── model_serving.py # Model serving utilities
│   │   ├── reports.py      # Report generation
│   │   └── utils.py        # Utility functions
│   ├── reports/             # Report templates and temporary files
│   │   ├── header.html     # HTML report header template
│   │   └── temp/           # Temporary report files (cleaned up automatically)
│   ├── metrics.py           # Prometheus metrics definitions
│   └── __init__.py
├── scripts/                 # Test and utility scripts
│   ├── run_cleanup_test.py  # Manual cleanup workflow test
│   └── run_training_test.py # Training test with scenario support (--all for batch)
├── tests/                   # Test suite
│   ├── activities/          # Activity tests
│   ├── workflows/           # Workflow tests
│   ├── utils/               # Utility tests
│   ├── worker/              # Worker tests
│   ├── schedules/           # Schedule tests
│   └── sientia/             # Sientia module tests
├── docs/                    # Documentation and test data
│   ├── test-scenarios/      # JSON test scenario files for integration tests
│   └── test-model-data.csv  # Sample CSV data for testing
├── .github/workflows/       # CI/CD workflows
│   ├── quality-gate.yml     # PR quality checks
│   └── deploy.yml           # Deployment workflow
├── Dockerfile               # Container image definition
├── values.yaml              # Helm chart values
├── pyproject.toml           # Project configuration
├── requirements.txt         # Production dependencies
├── requirements-dev.txt     # Development dependencies
├── validate.sh              # Code quality validation script
├── run_local.sh             # Local execution script
└── README.md                # This file

Adding New Features

1. Follow Temporal patterns for new workflows and activities
2. Add comprehensive docstrings for all public methods
3. Include Prometheus metrics for monitoring
4. Add unit tests for new functionality (maintain 80%+ coverage)
5. Run validation script (./validate.sh) before committing
6. Update this README with new features and configuration

Test Coverage Guidelines

• Minimum coverage: 80% (enforced by CI/CD)
• Current coverage: 100% 🎯
• Test all branches: Use Coverage Gutters to identify uncovered lines
• Mock external dependencies: Use unittest.mock for external services
• Async testing: Use pytest-asyncio for async activities and workflows
• Test structure:

  tests/
  ├── activities/      # Activity tests
  ├── workflows/       # Workflow tests
  ├── utils/           # Utility tests
  ├── worker/          # Worker tests
  ├── schedules/       # Schedule tests
  └── sientia/         # Sientia module tests
  

Troubleshooting

Common Issues

1. Temporal Connection Failures

   • Verify Temporal server is running and accessible
   • Check namespace configuration and permissions
   • Review server logs for connection issues

2. MLFlow Connection Issues

   • Verify MLFlow server is running and accessible
   • Check authentication credentials and permissions
   • Ensure model names and versions exist

3. Database Connection Issues

   • Verify PostgreSQL service is running
   • Check connection credentials and network access
   • Ensure proper connection pool configuration

4. Workflow Execution Failures

   • Review activity error logs and notifications
   • Check training parameter validation errors
   • Verify input data format and required fields

Debug Mode

Enable debug logging by setting the log level:

export LOG_LEVEL=DEBUG

Performance Tuning

Key Parameters

• Worker Concurrency: Adjust max_concurrent_workflow_tasks and max_concurrent_activities
  • Training workflows: 10 concurrent tasks/activities
  • Cleanup workflows: 20 concurrent tasks/activities
• Connection Pools: Optimize database connection pool sizes
• Model Retention: Configure MLFlow model retention based on requirements
• Batch Sizes: Adjust data processing batch sizes for optimal throughput
• Cleanup Performance: Tune MAX_KEYS_CLEANUP for MinIO list operation page size

Scaling Considerations

• Horizontal Scaling: Deploy multiple worker instances
• Task Queue Distribution: Two dedicated task queues for workflow isolation
  • train_model-queue: Training workflows
  • cleanup-queue: Cleanup workflows
• Database Performance: Optimize indexes and connection pooling
• MLFlow Performance: Configure appropriate model serving resources
• Storage Management: Adjust cleanup retention period based on storage capacity and costs

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes with comprehensive testing
4. Update documentation and docstrings
5. Submit a pull request

Code Quality Standards

• Follow PEP 8 style guidelines
• Include comprehensive docstrings for all public methods
• Maintain test coverage above 80%
• Use type hints where appropriate
• Follow Temporal.io best practices

License

This project is licensed under the terms specified in the LICENSE file.

Support

For support and questions:

• Check the troubleshooting section above
• Review the metrics and logs for error patterns
• Open an issue in the project repository
• Contact the development team

Local GitHub Actions Testing (act)

What is act?

act is a tool that allows you to run GitHub Actions locally using Docker. This is useful for:

• Testing workflows before pushing to the repository
• Debugging issues in workflows without creating commits
• Speeding up development by avoiding push/wait/check cycles
• Saving GitHub Actions minutes during development

Installation

Prerequisites

• Docker installed and running
• Go (for installation via go install)

Installation Steps

# 1. Update packages
sudo apt-get update

# 2. Install Go (if not already installed)
sudo apt-get install golang

# 3. Install act
go install github.com/nektos/act@latest

# 4. Add Go bin to PATH
echo 'export PATH="$PATH:$HOME/go/bin"' >> ~/.bashrc
source ~/.bashrc

# 5. Verify installation
act --version

On first run, act will ask which Docker image to use:

• Large (~17GB): Full image, compatible with almost all actions
• Medium (~500MB): Balanced image, compatible with most actions ✅ Recommended
• Micro (<200MB): Minimal image, Node.js only

Configuration

.secrets File

Create a .secrets file in the project root to store tokens and credentials:

# .secrets
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx
SONAR_TOKEN=sqp_xxxxxxxxxxxxxxxxxxxx
SONAR_HOST_URL=https://sonarqube.example.com
CI_DEPS_APP_ID=123456
CI_DEPS_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"

⚠️ Important: The .secrets file is already in .gitignore. Never commit this file!

.event.json File

Create a .event.json file to simulate GitHub events (e.g., pull request):

{
  "pull_request": {
    "head": {
      "ref": "feature/my-feature"
    },
    "number": 1
  }
}

⚠️ Important: The .event.json file is already in .gitignore. Never commit this file!

Usage

List Available Jobs

act -l

This command lists all available workflows and jobs in the repository.

Run Quality Gate Locally

act pull_request -j quality-gate \
  --secret-file .secrets \
  --env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true" \
  --eventpath .event.json

Run Deploy Workflow with Local Repository

If you have workflows that reference external repositories (e.g., reusable workflows), you can map them locally:

act pull_request \
  -e .event.json \
  --secret-file .secrets \
  -W .github/workflows/deploy.yml \
  --local-repository Aignosi/github_workflow_templates=/path/to/local/github_workflow_templates \
  --container-daemon-socket /var/run/docker.sock \
  --container-options "--user $(id -u):$(id -g)"

Docker Socket Permissions

If you encounter permission issues with Docker socket:

# Grant temporary access to Docker socket
sudo chmod 666 /var/run/docker.sock

# Fix file ownership after running act (if needed)
sudo chown -R $USER:$USER /path/to/project

Command Reference

Command Parameters

Parameter	Description
pull_request	Event type to simulate (can be push, pull_request, workflow_dispatch, etc.)
-j quality-gate	Specific job name to execute (use act -l to see available jobs)
--secret-file .secrets	File containing secrets (tokens, credentials)
--env VAR=value	Sets environment variables for execution
--eventpath .event.json	JSON file with the simulated event payload

Special Parameter: SONAR_SCANNER_OPTS

--env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true"

This parameter is required because SonarQube tries to automatically detect the CI environment. When running locally with act, the complete GitHub Actions context is not available, causing errors. The -Dsonar.ci.autoconfig.disabled=true flag disables this automatic detection.

Other Useful Commands

# List all jobs
act -l

# Run with verbose output
act pull_request -j quality-gate --secret-file .secrets -v

# Run a push event
act push -j build --secret-file .secrets

# Use a specific Docker image
act -P ubuntu-latest=catthehacker/ubuntu:act-latest

# Dry-run (doesn't execute, only shows what would be done)
act -n

Troubleshooting

Problem	Solution
SyntaxError: Unexpected end of JSON input	Check if .event.json is properly formatted
NullPointerException in SonarQube	Add --env SONAR_SCANNER_OPTS="-Dsonar.ci.autoconfig.disabled=true"
Not Found when accessing GitHub API	Check if GITHUB_TOKEN in .secrets is valid
Job not found	Use act -l to see the correct job names

Docker

Create image

$ docker build --ssh default --no-cache --progress=plain -t aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0 .

Create container

$ docker run --env-file .env --network="host" --name sientia-dataops-model-manager -d aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0

$ docker logs -f sientia-dataops-model-manager

Login using access token

$ docker login -u <username> -p <access-token> aignosi.azurecr.io

Push image to repository

$ docker push aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0

Helm Chart

Reference

https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/274563074/Como+utilizar+o+Helm+Repo+Privado

Add Helm Chart repository

$ helm repo add sientia \
    https://raw.githubusercontent.com/Aignosi/sientia-dataops-helm-repo/refs/heads/main/ \
    --username $GITHUB_USER \
    --password $GITHUB_PASS

# Update repository
$ helm repo update

# List repositories
$ helm repo list

# List versions of a specific chart
$ helm search repo sientia --versions

# List all charts available
$ helm search repo sientia

# List chart details
$ helm show all sientia/sientia-module

# Download chart to current directory
$ helm pull sientia/sientia-module --version 0.6.0 --untar

# Remove chart directory
$ rm -rf sientia-module

Helm Install

$ helm upgrade --install sientia-dataops-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.0

Uninstall Helm Chart

$ helm uninstall sientia-dataops-model-manager -n sientia

---

Note: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.