SIENTIAPDE-1248: Refactor README.md to include detailed documentation on features, architecture, workflows, installation, code quality, testing, monitoring, configuration, development, troubleshooting, performance tuning, contributing, license, and support.

2025-10-04 14:45:42 -03:00
parent 0db7f7462a
commit ec5b9df125
1 changed files with 300 additions and 96 deletions
--- a/README.md
+++ b/README.md
@@ -2,6 +2,67 @@

 A comprehensive AI model management platform for the complete machine learning lifecycle. Handles model training, versioning, deployment, monitoring, and governance. Streamlines MLOps workflows with centralized model registry, automated pipelines, performance tracking, and enterprise-grade compliance features.

+## 📑 Table of Contents
+
+- [Features](#features)
+  - [Core Functionality](#core-functionality)
+  - [Advanced Capabilities](#advanced-capabilities)
+  - [Development & Quality Assurance](#development--quality-assurance)
+- [Architecture](#architecture)
+  - [Architecture Principles](#architecture-principles)
+  - [Key Components](#key-components)
+  - [Data Flow Architecture](#data-flow-architecture)
+  - [Security Architecture](#security-architecture)
+- [Workflows](#workflows)
+  - [Predictions Batch Workflow](#1-predictions-batch-workflow-predictions_batchpy)
+  - [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
+  - [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
+  - [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
+- [Installation & Setup](#installation--setup)
+  - [Prerequisites](#prerequisites)
+  - [Environment Setup](#environment-setup)
+  - [Temporal Namespace Setup](#temporal-namespace-setup)
+  - [Local Development Setup](#local-development-setup)
+- [How to Run](#how-to-run)
+  - [Running the Model Manager Application](#running-the-model-manager-application)
+  - [Running Tests and Coverage](#running-tests-and-coverage)
+  - [Manual Test Execution](#manual-test-execution)
+  - [Manual Application Execution](#manual-application-execution)
+- [Code Quality & Validation](#code-quality--validation)
+  - [Overview](#overview)
+  - [Validation Tools](#validation-tools)
+  - [Tools Installation](#tools-installation)
+  - [Complete Validation](#complete-validation)
+  - [Automatic Fixes](#automatic-fixes)
+  - [Configuration](#configuration)
+  - [CI/CD Integration](#cicd-integration)
+  - [Best Practices](#best-practices)
+- [Testing](#testing)
+  - [Test Structure](#test-structure)
+  - [Test Execution](#test-execution)
+- [Monitoring and Metrics](#monitoring-and-metrics)
+  - [Application Health Metrics](#application-health-metrics)
+  - [Prediction Operation Metrics](#prediction-operation-metrics)
+  - [Data Quality Metrics](#data-quality-metrics)
+- [Configuration](#configuration-1)
+  - [Environment Variables](#environment-variables)
+  - [Workflow Configuration](#workflow-configuration)
+- [Development](#development)
+  - [Code Quality & Testing](#code-quality--testing)
+  - [Project Structure](#project-structure)
+  - [Adding New Features](#adding-new-features)
+  - [Test Coverage Guidelines](#test-coverage-guidelines)
+- [Troubleshooting](#troubleshooting)
+  - [Common Issues](#common-issues)
+  - [Debug Mode](#debug-mode)
+- [Performance Tuning](#performance-tuning)
+  - [Key Parameters](#key-parameters)
+  - [Scaling Considerations](#scaling-considerations)
+- [Contributing](#contributing)
+  - [Code Quality Standards](#code-quality-standards)
+- [License](#license)
+- [Support](#support)
+
 ## Features

 ### Core Functionality
@@ -21,9 +82,11 @@ A comprehensive AI model management platform for the complete machine learning l

 ### Development & Quality Assurance
 - **Code Quality Tools**: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
- **Automated Validation**: Pre-commit validation script and CI/CD integration
- **Comprehensive Testing**: pytest with async support and 70%+ code coverage
+- **Automated Validation**: Pre-commit validation script (`validate.sh`) and CI/CD integration
+- **Comprehensive Testing**: pytest with async support and **99%+ code coverage** 🎯
 - **Type Safety**: Static type checking with mypy for improved code reliability
+- **Coverage Visualization**: Integration with Coverage Gutters for real-time coverage feedback
+- **Automated Versioning**: Semantic versioning based on branch patterns (release/*, feature/*, fix/*, rc/*)

 ## Architecture

@@ -142,7 +205,7 @@ Production Update → Notification & Monitoring
 - **Audit Logging**: Comprehensive access and operation logging
 - **Data Retention**: Configurable data lifecycle management

-## 🔄 Workflows
+## Workflows

 ### 1. Predictions Batch Workflow (`predictions_batch.py`)

@@ -331,12 +394,15 @@ flowchart LR
    D -.-> PostgreSQL[(PostgreSQL)]
 ```

-## 📋 Prerequisites
+## Installation & Setup
+
+### Prerequisites

 - Python 3.11+
 - Temporal server/cluster
 - PostgreSQL database
 - MLFlow server
+- MinIO object storage (for MLFlow artifacts)
 - MongoDB server (for notifications)

 **Note**: External dependencies must be available either through:
@@ -345,6 +411,44 @@ flowchart LR
 - Cloud-managed services
 - Local installations

+#### MinIO Setup
+
+MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:
+
+📚 **[Install MinIO via Helm Chart on K8s](https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/225214465/Install+Minio+via+Helm+Chart+on+K8s)**
+
+This guide covers:
+- Helm chart installation on Kubernetes
+- Storage configuration and persistence
+- Access credentials setup
+- Integration with MLFlow
+
+### Environment Setup
+
+1. **Clone the repository**:
+   ```bash
+   git clone <repository-url>
+   cd sientia-dataops-model-manager
+   ```
+
+2. **Create virtual environment**:
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+3. **Install dependencies**:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+4. **Configure environment variables** (see [Configuration](#-configuration) section)
+
+5. **Run validation script**:
+   ```bash
+   ./validate.sh
+   ```
+
 ### Temporal Namespace Setup

 The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created **before** starting the application.
@@ -448,8 +552,6 @@ kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
 - Contact cluster administrator for namespace creation
 - Or request elevated permissions for your service account

-## 🚀 Installation
-
 ### Local Development Setup

 1. **Clone the repository**
@@ -512,7 +614,7 @@ kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
   # Ensure services are accessible on localhost with appropriate ports
   ```

-## 📦 How to Run
+## How to Run

 ### Running the Model Manager Application

@@ -585,122 +687,123 @@ fi
 python -m model_manager.worker.worker
 ```

-## 🔍 Code Quality & Validation
+## Code Quality & Validation

 ### Overview

-Como Python não é uma linguagem compilada, utilizamos um conjunto robusto de ferramentas para validar a qualidade, segurança e correção do código antes da execução. Estas ferramentas detectam erros, problemas de estilo, vulnerabilidades de segurança e garantem a consistência do código.
+Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.

-### Ferramentas de Validação
+### Validation Tools

-#### 1. **Ruff** - Linting e Formatação ⚡
-Ferramenta moderna e extremamente rápida (escrita em Rust) que substitui múltiplas ferramentas:
- **Linting**: Detecta erros de código, problemas de estilo (PEP 8), bugs comuns
- **Formatação**: Formata código automaticamente de forma consistente
- **Velocidade**: 10-100x mais rápido que Flake8/Black
+#### 1. **Ruff** - Linting and Formatting ⚡
+Modern and extremely fast tool (written in Rust) that replaces multiple tools:
+- **Linting**: Detects code errors, style issues (PEP 8), common bugs
+- **Formatting**: Automatically formats code consistently
+- **Speed**: 10-100x faster than Flake8/Black

 #### 2. **mypy** - Type Checking 🏷️
-Verificador de tipos estáticos que analisa type hints:
- Detecta erros de tipo antes da execução
- Melhora a documentação do código
- Previne bugs relacionados a tipos incorretos
+Static type checker that analyzes type hints:
+- Detects type errors before execution
+- Improves code documentation
+- Prevents bugs related to incorrect types

-#### 3. **Bandit** - Análise de Segurança 🔒
-Scanner de vulnerabilidades de segurança:
- Detecta padrões inseguros de código
- Identifica hardcoded passwords, SQL injection, etc.
- Garante conformidade com práticas de segurança
+#### 3. **Bandit** - Security Analysis 🔒
+Security vulnerability scanner:
+- Detects insecure code patterns
+- Identifies hardcoded passwords, SQL injection, etc.
+- Ensures compliance with security practices

-#### 4. **pytest** - Testes Automatizados 🧪
-Framework de testes com cobertura de código:
- Executa testes unitários e de integração
- Mede cobertura de código
- Suporta testes assíncronos
+#### 4. **pytest** - Automated Testing 🧪
+Testing framework with code coverage:
+- Executes unit and integration tests
+- Measures code coverage
+- Supports asynchronous tests

-### Instalação das Ferramentas
+### Tools Installation

 ```bash
-# Instalar dependências de desenvolvimento
+# Install development dependencies
 pip install -r requirements-dev.txt
 ```

-### Validação Completa
+### Complete Validation

-#### Opção 1: Script Automatizado (Recomendado)
+#### Option 1: Automated Script (Recommended)
 ```bash
-# Executar todas as validações de uma vez
+# Run all validations at once
 ./validate.sh
 ```

-O script `validate.sh` executa automaticamente:
-1. ✅ Verificação de formatação (Ruff)
-2. ✅ Linting de código (Ruff)
+The `validate.sh` script automatically executes:
+1. ✅ Format checking (Ruff)
+2. ✅ Code linting (Ruff)
 3. ✅ Type checking (mypy)
-4. ✅ Análise de segurança (Bandit)
-5. ✅ Testes unitários com cobertura (pytest)
+4. ✅ Security analysis (Bandit)
+5. ✅ Unit tests with coverage (pytest)

-#### Opção 2: Comandos Individuais
+#### Option 2: Individual Commands
 ```bash
-# 1. Verificar formatação
+# 1. Check formatting
 ruff format --check model_manager/ tests/

-# 2. Verificar linting
+# 2. Check linting
 ruff check model_manager/ tests/

-# 3. Verificar tipos
+# 3. Check types
 mypy model_manager/

-# 4. Análise de segurança
+# 4. Security analysis
 bandit -r model_manager/ -ll

-# 5. Executar testes
+# 5. Run tests
 pytest tests/ --cov=model_manager --cov-report=term-missing
 ```

-### Correção Automática
+### Automatic Fixes

-Algumas ferramentas podem corrigir problemas automaticamente:
+Some tools can automatically fix issues:

 ```bash
-# Formatar código automaticamente
+# Format code automatically
 ruff format model_manager/ tests/

-# Corrigir problemas de linting automaticamente
+# Fix linting issues automatically
 ruff check --fix model_manager/ tests/
 ```

-### Configuração
+### Configuration

-Todas as ferramentas são configuradas no arquivo `pyproject.toml`:
- **Ruff**: Regras de linting, formatação, complexidade
- **mypy**: Configurações de type checking
- **pytest**: Opções de teste e cobertura
- **Bandit**: Regras de segurança
+All tools are configured in the `pyproject.toml` file:
+- **Ruff**: Linting rules, formatting, complexity
+- **mypy**: Type checking settings
+- **pytest**: Test and coverage options
+- **Bandit**: Security rules

-### Integração com CI/CD
+### CI/CD Integration

-O workflow `.github/workflows/quality-gate.yml` executa automaticamente todas as validações em cada push/PR:
- ✅ Formatação e linting bloqueiam merge se falharem
- ⚠️ Type checking e segurança geram avisos mas não bloqueiam
- ✅ Testes devem passar com cobertura mínima de 70%
+The `.github/workflows/quality-gate.yml` workflow automatically runs all validations on each push/PR:
+- ✅ Formatting and linting block merge if they fail
+- ⚠️ Type checking and security generate warnings but don't block
+- ✅ Tests must pass with minimum 80% coverage

-### Boas Práticas
+### Best Practices

-1. **Antes de Commit**: Execute `./validate.sh` para garantir qualidade
-2. **Durante Desenvolvimento**: Use `ruff check --watch` para feedback em tempo real
-3. **Type Hints**: Adicione type hints em funções novas para melhor validação
-4. **Testes**: Mantenha cobertura acima de 70%
-5. **Segurança**: Revise e corrija todos os avisos do Bandit
+1. **Before Commit**: Run `./validate.sh` to ensure quality
+2. **During Development**: Use `ruff check --watch` for real-time feedback
+3. **Type Hints**: Add type hints to new functions for better validation
+4. **Tests**: Maintain coverage above 80%
+5. **Security**: Review and fix all Bandit warnings

-## 🧪 Testing
+## Testing

 ### Test Structure
 ```
 tests/
-├── activities/           # Activity implementation tests
-├── workflow/            # Workflow orchestration tests
-├── utils/               # Utility function tests
-└── integration/         # End-to-end workflow tests
+├── laborious/
+│   ├── activities/      # Activity implementation tests
+│   ├── workflows/       # Workflow orchestration tests
+│   ├── utils/          # Utility function tests
+│   └── worker/         # Worker tests
 ```

 ### Test Execution
@@ -712,11 +815,11 @@ pip install pytest pytest-cov pytest-asyncio
 pytest --cov=model_manager --cov-report=html

 # Run specific test modules
-pytest tests/activities/test_gates.py
-pytest tests/workflow/test_predictions_batch.py
+pytest tests/laborious/activities/test_gates.py
+pytest tests/laborious/workflows/test_predictions_batch.py
 ```

-## 📊 Monitoring and Metrics
+## Monitoring and Metrics

 The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:

@@ -738,7 +841,7 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 - MLFlow API response validation metrics
 - Data quality gate performance tracking

-## ⚙️ Configuration
+## Configuration

 ### Environment Variables

@@ -757,6 +860,15 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 | `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
 | `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
 | `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
+| `MINIO_ENDPOINT_URL` | MinIO server endpoint | `http://minio.minio.svc.cluster.local:9000` | Yes |
+| `MINIO_ACCESS_KEY` | MinIO access key | `minioadmin` | Yes |
+| `MINIO_SECRET_KEY` | MinIO secret key | `minioadmin` | Yes |
+| `MINIO_REGION` | MinIO region | `us-east-1` | No |
+| `MINIO_USE_SSL` | Enable SSL for MinIO | `false` | No |
+| `MINIO_MAX_RETRY_ATTEMPTS` | Maximum retry attempts | `3` | No |
+| `MINIO_RETRY_MODE` | Retry mode (standard/adaptive) | `adaptive` | No |
+| `MINIO_CONNECT_TIMEOUT` | Connection timeout (seconds) | `10` | No |
+| `MINIO_READ_TIMEOUT` | Read timeout (seconds) | `60` | No |
 | `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
 | `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
 | `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
@@ -856,30 +968,104 @@ This is the configuration created by the Orchestrator in Temporal.
 }
 ```

-## 🔧 Development
+## Development
+
+### Code Quality & Testing
+
+The project maintains **99%+ code coverage** with comprehensive unit and integration tests.
+
+#### Running Tests
+
+```bash
+# Run all tests with coverage
+pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html
+
+# Run specific test file
+pytest tests/laborious/activities/test_gates.py -v
+
+# Run with coverage visualization
+pytest tests/ --cov=model_manager --cov-report=xml
+# Then open htmlcov/index.html in browser
+```
+
+#### Validation Script
+
+The `validate.sh` script runs all quality checks before commit:
+
+```bash
+./validate.sh
+```
+
+This script performs:
+1. ✅ **Code Formatting** (Ruff)
+2. ✅ **Code Linting** (Ruff)
+3. ✅ **Type Checking** (mypy)
+4. ✅ **Security Analysis** (Bandit)
+5. ✅ **Unit Tests** (pytest with 80%+ coverage requirement)
+
+#### Coverage Visualization
+
+For real-time coverage feedback in VS Code/Windsurf:
+
+1. **Install Coverage Gutters extension**
+2. **Configure `.vscode/settings.json`**:
+   ```json
+   {
+     "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
+     "coverage-gutters.coverageFileNames": ["coverage.xml"],
+     "coverage-gutters.showLineCoverage": true,
+     "coverage-gutters.showRulerCoverage": true
+   }
+   ```
+3. **Run tests to generate coverage**:
+   ```bash
+   pytest tests/ --cov=model_manager --cov-report=xml
+   ```
+4. **Activate Coverage Gutters**: Press `Ctrl+Shift+7` (or `Cmd+Shift+7` on Mac)
+
+#### Automated Versioning
+
+The project uses semantic versioning based on branch patterns:
+
+| Branch Pattern | Version Change | Example |
+|---------------|----------------|---------|
+| `release/*` | Major version bump | `2.0.0` |
+| `feature/*` | Minor version bump | `1.2.0` |
+| `fix/*` | Patch version bump | `1.1.3` |
+| `rc/*` | Release candidate | `1.1.2-rc2` |
+
+Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.

 ### Project Structure
 ```
 model_manager/
 ├── activities/              # Temporal activity implementations
-│   ├── activities.py       # Main activities orchestrator
-│   ├── gates.py            # Data quality gates and filtering
-│   └── mlflow.py           # MLFlow model operations
+│   ├── __init__.py
+│   ├── activities.py       # Main activities orchestrator (combines all activities)
+│   ├── gates.py            # Data quality gates and filtering logic
+│   ├── minio.py            # MinIO object storage operations
+│   └── mlflow.py           # MLFlow model operations (predict/transform)
 ├── workflows/               # Temporal workflow definitions
-│   ├── predictions_batch.py # Main batch prediction workflow
+│   ├── __init__.py
+│   ├── predictions_batch.py # Main batch prediction workflow entry point
 │   ├── minimal_retrain.py  # Model retraining workflow
 │   └── sub_workflows/      # Sub-workflow implementations
-│       ├── prediction_process.py # Core prediction workflow
-│       └── format_and_export_prediction.py # Export workflow
+│       ├── __init__.py
+│       ├── prediction_process.py # Core prediction pipeline
+│       └── format_and_export_prediction.py # Data export workflow
 ├── worker/                  # Worker implementation
-│   └── worker.py           # Main worker orchestrator
-├── utils/                   # Utility functions
-│   ├── connectors_config.py # Database configuration
-│   ├── filters/            # Data quality filters
-│   │   ├── conditional_filters.py # Conditional data filters
-│   │   └── mlflow_filters.py # MLFlow response filters
+│   ├── __init__.py
+│   └── worker.py           # Main worker orchestrator (Temporal client setup)
+├── utils/                   # Utility functions and helpers
+│   ├── __init__.py
+│   ├── connectors_config.py # Environment-based configuration builders
+│   ├── filters/            # Data quality validation filters
+│   │   ├── __init__.py
+│   │   ├── conditional_filters.py # Input data validation filters
+│   │   └── mlflow_filters.py # MLFlow response validation filters
 │   └── repository/         # Data access layer
-│       └── model_repository.py # MLFlow model operations
+│       ├── __init__.py
+│       └── model_repository.py # MLFlow model operations and retraining
 ├── metrics.py               # Prometheus metrics definitions
 └── __init__.py
 ```
@@ -889,10 +1075,28 @@ model_manager/
 1. **Follow Temporal patterns** for new workflows and activities
 2. **Add comprehensive docstrings** for all public methods
 3. **Include Prometheus metrics** for monitoring
-4. **Add unit tests** for new functionality
-5. **Update this README** with new features and configuration
+4. **Add unit tests** for new functionality (maintain 80%+ coverage)
+5. **Run validation script** (`./validate.sh`) before committing
+6. **Update this README** with new features and configuration

-## 🐛 Troubleshooting
+### Test Coverage Guidelines
+
+- **Minimum coverage**: 80% (enforced by CI/CD)
+- **Current coverage**: 99%+ 🎯
+- **Test all branches**: Use Coverage Gutters to identify uncovered lines
+- **Mock external dependencies**: Use `unittest.mock` for external services
+- **Async testing**: Use `pytest-asyncio` for async activities and workflows
+- **Test structure**:
+  ```
+  tests/
+  ├── laborious/
+  │   ├── activities/      # Activity tests
+  │   ├── workflows/       # Workflow tests
+  │   ├── utils/          # Utility tests
+  │   └── worker/         # Worker tests
+  ```
+
+## Troubleshooting

 ### Common Issues

@@ -923,7 +1127,7 @@ Enable debug logging by setting the log level:
 export LOG_LEVEL=DEBUG
 ```

-## ⚡ Performance Tuning
+## Performance Tuning

 ### Key Parameters

@@ -939,7 +1143,7 @@ export LOG_LEVEL=DEBUG
 - **Database Performance**: Optimize indexes and connection pooling
 - **MLFlow Performance**: Configure appropriate model serving resources

-## 🤝 Contributing
+## Contributing

 1. Fork the repository
 2. Create a feature branch
@@ -955,11 +1159,11 @@ export LOG_LEVEL=DEBUG
 - Use type hints where appropriate
 - Follow Temporal.io best practices

-## 📄 License
+## License

 This project is licensed under the terms specified in the LICENSE file.

-## 🆘 Support
+## Support

 For support and questions:
 - Check the troubleshooting section above