SIENTIAPDE-1248: Refactor README.md to include detailed documentation on features, architecture, workflows, installation, code quality, testing, monitoring, configuration, development, troubleshooting, performance tuning, contributing, license, and support.

README.md

@@ -2,6 +2,67 @@
 
 A comprehensive AI model management platform for the complete machine learning lifecycle. Handles model training, versioning, deployment, monitoring, and governance. Streamlines MLOps workflows with centralized model registry, automated pipelines, performance tracking, and enterprise-grade compliance features.
 
+## 📑 Table of Contents
+
+- [Features](#features)
+  - [Core Functionality](#core-functionality)
+  - [Advanced Capabilities](#advanced-capabilities)
+  - [Development & Quality Assurance](#development--quality-assurance)
+- [Architecture](#architecture)
+  - [Architecture Principles](#architecture-principles)
+  - [Key Components](#key-components)
+  - [Data Flow Architecture](#data-flow-architecture)
+  - [Security Architecture](#security-architecture)
+- [Workflows](#workflows)
+  - [Predictions Batch Workflow](#1-predictions-batch-workflow-predictions_batchpy)
+  - [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
+  - [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
+  - [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
+- [Installation & Setup](#installation--setup)
+  - [Prerequisites](#prerequisites)
+  - [Environment Setup](#environment-setup)
+  - [Temporal Namespace Setup](#temporal-namespace-setup)
+  - [Local Development Setup](#local-development-setup)
+- [How to Run](#how-to-run)
+  - [Running the Model Manager Application](#running-the-model-manager-application)
+  - [Running Tests and Coverage](#running-tests-and-coverage)
+  - [Manual Test Execution](#manual-test-execution)
+  - [Manual Application Execution](#manual-application-execution)
+- [Code Quality & Validation](#code-quality--validation)
+  - [Overview](#overview)
+  - [Validation Tools](#validation-tools)
+  - [Tools Installation](#tools-installation)
+  - [Complete Validation](#complete-validation)
+  - [Automatic Fixes](#automatic-fixes)
+  - [Configuration](#configuration)
+  - [CI/CD Integration](#cicd-integration)
+  - [Best Practices](#best-practices)
+- [Testing](#testing)
+  - [Test Structure](#test-structure)
+  - [Test Execution](#test-execution)
+- [Monitoring and Metrics](#monitoring-and-metrics)
+  - [Application Health Metrics](#application-health-metrics)
+  - [Prediction Operation Metrics](#prediction-operation-metrics)
+  - [Data Quality Metrics](#data-quality-metrics)
+- [Configuration](#configuration-1)
+  - [Environment Variables](#environment-variables)
+  - [Workflow Configuration](#workflow-configuration)
+- [Development](#development)
+  - [Code Quality & Testing](#code-quality--testing)
+  - [Project Structure](#project-structure)
+  - [Adding New Features](#adding-new-features)
+  - [Test Coverage Guidelines](#test-coverage-guidelines)
+- [Troubleshooting](#troubleshooting)
+  - [Common Issues](#common-issues)
+  - [Debug Mode](#debug-mode)
+- [Performance Tuning](#performance-tuning)
+  - [Key Parameters](#key-parameters)
+  - [Scaling Considerations](#scaling-considerations)
+- [Contributing](#contributing)
+  - [Code Quality Standards](#code-quality-standards)
+- [License](#license)
+- [Support](#support)
+
 ## Features
 
 ### Core Functionality
@@ -21,9 +82,11 @@ A comprehensive AI model management platform for the complete machine learning l
 
 ### Development & Quality Assurance
 - **Code Quality Tools**: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
-- **Automated Validation**: Pre-commit validation script and CI/CD integration
+- **Automated Validation**: Pre-commit validation script (`validate.sh`) and CI/CD integration
-- **Comprehensive Testing**: pytest with async support and 70%+ code coverage
+- **Comprehensive Testing**: pytest with async support and **99%+ code coverage** 🎯
 - **Type Safety**: Static type checking with mypy for improved code reliability
+- **Coverage Visualization**: Integration with Coverage Gutters for real-time coverage feedback
+- **Automated Versioning**: Semantic versioning based on branch patterns (release/*, feature/*, fix/*, rc/*)
 
 ## Architecture
 
@@ -142,7 +205,7 @@ Production Update → Notification & Monitoring
 - **Audit Logging**: Comprehensive access and operation logging
 - **Data Retention**: Configurable data lifecycle management
 
-## 🔄 Workflows
+## Workflows
 
 ### 1. Predictions Batch Workflow (`predictions_batch.py`)
 
@@ -331,12 +394,15 @@ flowchart LR
     D -.-> PostgreSQL[(PostgreSQL)]
 ```
 
-## 📋 Prerequisites
+## Installation & Setup
+
+### Prerequisites
 
 - Python 3.11+
 - Temporal server/cluster
 - PostgreSQL database
 - MLFlow server
+- MinIO object storage (for MLFlow artifacts)
 - MongoDB server (for notifications)
 
 **Note**: External dependencies must be available either through:
@@ -345,6 +411,44 @@ flowchart LR
 - Cloud-managed services
 - Local installations
 
+#### MinIO Setup
+
+MinIO is required for MLFlow artifact storage. For detailed installation and configuration instructions, refer to:
+
+📚 **[Install MinIO via Helm Chart on K8s](https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/225214465/Install+Minio+via+Helm+Chart+on+K8s)**
+
+This guide covers:
+- Helm chart installation on Kubernetes
+- Storage configuration and persistence
+- Access credentials setup
+- Integration with MLFlow
+
+### Environment Setup
+
+1. **Clone the repository**:
+   ```bash
+   git clone <repository-url>
+   cd sientia-dataops-model-manager
+   ```
+
+2. **Create virtual environment**:
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+3. **Install dependencies**:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+4. **Configure environment variables** (see [Configuration](#-configuration) section)
+
+5. **Run validation script**:
+   ```bash
+   ./validate.sh
+   ```
+
 ### Temporal Namespace Setup
 
 The Model Manager requires a dedicated Temporal namespace to isolate workflows and maintain proper execution history. The namespace must be created **before** starting the application.
@@ -448,8 +552,6 @@ kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
 - Contact cluster administrator for namespace creation
 - Or request elevated permissions for your service account
 
-## 🚀 Installation
-
 ### Local Development Setup
 
 1. **Clone the repository**
@@ -512,7 +614,7 @@ kubectl exec -n temporal <temporal-admin-tools-pod-name> -- \
    # Ensure services are accessible on localhost with appropriate ports
    ```
 
-## 📦 How to Run
+## How to Run
 
 ### Running the Model Manager Application
 
@@ -585,122 +687,123 @@ fi
 python -m model_manager.worker.worker
 ```
 
-## 🔍 Code Quality & Validation
+## Code Quality & Validation
 
 ### Overview
 
-Como Python não é uma linguagem compilada, utilizamos um conjunto robusto de ferramentas para validar a qualidade, segurança e correção do código antes da execução. Estas ferramentas detectam erros, problemas de estilo, vulnerabilidades de segurança e garantem a consistência do código.
+Since Python is not a compiled language, we use a robust set of tools to validate code quality, security, and correctness before execution. These tools detect errors, style issues, security vulnerabilities, and ensure code consistency.
 
-### Ferramentas de Validação
+### Validation Tools
 
-#### 1. **Ruff** - Linting e Formatação ⚡
+#### 1. **Ruff** - Linting and Formatting ⚡
-Ferramenta moderna e extremamente rápida (escrita em Rust) que substitui múltiplas ferramentas:
+Modern and extremely fast tool (written in Rust) that replaces multiple tools:
-- **Linting**: Detecta erros de código, problemas de estilo (PEP 8), bugs comuns
+- **Linting**: Detects code errors, style issues (PEP 8), common bugs
-- **Formatação**: Formata código automaticamente de forma consistente
+- **Formatting**: Automatically formats code consistently
-- **Velocidade**: 10-100x mais rápido que Flake8/Black
+- **Speed**: 10-100x faster than Flake8/Black
 
 #### 2. **mypy** - Type Checking 🏷️
-Verificador de tipos estáticos que analisa type hints:
+Static type checker that analyzes type hints:
-- Detecta erros de tipo antes da execução
+- Detects type errors before execution
-- Melhora a documentação do código
+- Improves code documentation
-- Previne bugs relacionados a tipos incorretos
+- Prevents bugs related to incorrect types
 
-#### 3. **Bandit** - Análise de Segurança 🔒
+#### 3. **Bandit** - Security Analysis 🔒
-Scanner de vulnerabilidades de segurança:
+Security vulnerability scanner:
-- Detecta padrões inseguros de código
+- Detects insecure code patterns
-- Identifica hardcoded passwords, SQL injection, etc.
+- Identifies hardcoded passwords, SQL injection, etc.
-- Garante conformidade com práticas de segurança
+- Ensures compliance with security practices
 
-#### 4. **pytest** - Testes Automatizados 🧪
+#### 4. **pytest** - Automated Testing 🧪
-Framework de testes com cobertura de código:
+Testing framework with code coverage:
-- Executa testes unitários e de integração
+- Executes unit and integration tests
-- Mede cobertura de código
+- Measures code coverage
-- Suporta testes assíncronos
+- Supports asynchronous tests
 
-### Instalação das Ferramentas
+### Tools Installation
 
 ```bash
-# Instalar dependências de desenvolvimento
+# Install development dependencies
 pip install -r requirements-dev.txt
 ```
 
-### Validação Completa
+### Complete Validation
 
-#### Opção 1: Script Automatizado (Recomendado)
+#### Option 1: Automated Script (Recommended)
 ```bash
-# Executar todas as validações de uma vez
+# Run all validations at once
 ./validate.sh
 ```
 
-O script `validate.sh` executa automaticamente:
+The `validate.sh` script automatically executes:
-1. ✅ Verificação de formatação (Ruff)
+1. ✅ Format checking (Ruff)
-2. ✅ Linting de código (Ruff)
+2. ✅ Code linting (Ruff)
 3. ✅ Type checking (mypy)
-4. ✅ Análise de segurança (Bandit)
+4. ✅ Security analysis (Bandit)
-5. ✅ Testes unitários com cobertura (pytest)
+5. ✅ Unit tests with coverage (pytest)
 
-#### Opção 2: Comandos Individuais
+#### Option 2: Individual Commands
 ```bash
-# 1. Verificar formatação
+# 1. Check formatting
 ruff format --check model_manager/ tests/
 
-# 2. Verificar linting
+# 2. Check linting
 ruff check model_manager/ tests/
 
-# 3. Verificar tipos
+# 3. Check types
 mypy model_manager/
 
-# 4. Análise de segurança
+# 4. Security analysis
 bandit -r model_manager/ -ll
 
-# 5. Executar testes
+# 5. Run tests
 pytest tests/ --cov=model_manager --cov-report=term-missing
 ```
 
-### Correção Automática
+### Automatic Fixes
 
-Algumas ferramentas podem corrigir problemas automaticamente:
+Some tools can automatically fix issues:
 
 ```bash
-# Formatar código automaticamente
+# Format code automatically
 ruff format model_manager/ tests/
 
-# Corrigir problemas de linting automaticamente
+# Fix linting issues automatically
 ruff check --fix model_manager/ tests/
 ```
 
-### Configuração
+### Configuration
 
-Todas as ferramentas são configuradas no arquivo `pyproject.toml`:
+All tools are configured in the `pyproject.toml` file:
-- **Ruff**: Regras de linting, formatação, complexidade
+- **Ruff**: Linting rules, formatting, complexity
-- **mypy**: Configurações de type checking
+- **mypy**: Type checking settings
-- **pytest**: Opções de teste e cobertura
+- **pytest**: Test and coverage options
-- **Bandit**: Regras de segurança
+- **Bandit**: Security rules
 
-### Integração com CI/CD
+### CI/CD Integration
 
-O workflow `.github/workflows/quality-gate.yml` executa automaticamente todas as validações em cada push/PR:
+The `.github/workflows/quality-gate.yml` workflow automatically runs all validations on each push/PR:
-- ✅ Formatação e linting bloqueiam merge se falharem
+- ✅ Formatting and linting block merge if they fail
-- ⚠️ Type checking e segurança geram avisos mas não bloqueiam
+- ⚠️ Type checking and security generate warnings but don't block
-- ✅ Testes devem passar com cobertura mínima de 70%
+- ✅ Tests must pass with minimum 80% coverage
 
-### Boas Práticas
+### Best Practices
 
-1. **Antes de Commit**: Execute `./validate.sh` para garantir qualidade
+1. **Before Commit**: Run `./validate.sh` to ensure quality
-2. **Durante Desenvolvimento**: Use `ruff check --watch` para feedback em tempo real
+2. **During Development**: Use `ruff check --watch` for real-time feedback
-3. **Type Hints**: Adicione type hints em funções novas para melhor validação
+3. **Type Hints**: Add type hints to new functions for better validation
-4. **Testes**: Mantenha cobertura acima de 70%
+4. **Tests**: Maintain coverage above 80%
-5. **Segurança**: Revise e corrija todos os avisos do Bandit
+5. **Security**: Review and fix all Bandit warnings
 
-## 🧪 Testing
+## Testing
 
 ### Test Structure
 ```
 tests/
-├── activities/           # Activity implementation tests
+├── laborious/
-├── workflow/            # Workflow orchestration tests
+│   ├── activities/      # Activity implementation tests
-├── utils/               # Utility function tests
+│   ├── workflows/       # Workflow orchestration tests
-└── integration/         # End-to-end workflow tests
+│   ├── utils/          # Utility function tests
+│   └── worker/         # Worker tests
 ```
 
 ### Test Execution
@@ -712,11 +815,11 @@ pip install pytest pytest-cov pytest-asyncio
 pytest --cov=model_manager --cov-report=html
 
 # Run specific test modules
-pytest tests/activities/test_gates.py
+pytest tests/laborious/activities/test_gates.py
-pytest tests/workflow/test_predictions_batch.py
+pytest tests/laborious/workflows/test_predictions_batch.py
 ```
 
-## 📊 Monitoring and Metrics
+## Monitoring and Metrics
 
 The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:
 
@@ -738,7 +841,7 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 - MLFlow API response validation metrics
 - Data quality gate performance tracking
 
-## ⚙️ Configuration
+## Configuration
 
 ### Environment Variables
 
@@ -757,6 +860,15 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 | `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
 | `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
 | `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
+| `MINIO_ENDPOINT_URL` | MinIO server endpoint | `http://minio.minio.svc.cluster.local:9000` | Yes |
+| `MINIO_ACCESS_KEY` | MinIO access key | `minioadmin` | Yes |
+| `MINIO_SECRET_KEY` | MinIO secret key | `minioadmin` | Yes |
+| `MINIO_REGION` | MinIO region | `us-east-1` | No |
+| `MINIO_USE_SSL` | Enable SSL for MinIO | `false` | No |
+| `MINIO_MAX_RETRY_ATTEMPTS` | Maximum retry attempts | `3` | No |
+| `MINIO_RETRY_MODE` | Retry mode (standard/adaptive) | `adaptive` | No |
+| `MINIO_CONNECT_TIMEOUT` | Connection timeout (seconds) | `10` | No |
+| `MINIO_READ_TIMEOUT` | Read timeout (seconds) | `60` | No |
 | `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
 | `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
 | `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
@@ -856,30 +968,104 @@ This is the configuration created by the Orchestrator in Temporal.
 }
 ```
 
-## 🔧 Development
+## Development
+
+### Code Quality & Testing
+
+The project maintains **99%+ code coverage** with comprehensive unit and integration tests.
+
+#### Running Tests
+
+```bash
+# Run all tests with coverage
+pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=xml --cov-report=html
+
+# Run specific test file
+pytest tests/laborious/activities/test_gates.py -v
+
+# Run with coverage visualization
+pytest tests/ --cov=model_manager --cov-report=xml
+# Then open htmlcov/index.html in browser
+```
+
+#### Validation Script
+
+The `validate.sh` script runs all quality checks before commit:
+
+```bash
+./validate.sh
+```
+
+This script performs:
+1. ✅ **Code Formatting** (Ruff)
+2. ✅ **Code Linting** (Ruff)
+3. ✅ **Type Checking** (mypy)
+4. ✅ **Security Analysis** (Bandit)
+5. ✅ **Unit Tests** (pytest with 80%+ coverage requirement)
+
+#### Coverage Visualization
+
+For real-time coverage feedback in VS Code/Windsurf:
+
+1. **Install Coverage Gutters extension**
+2. **Configure `.vscode/settings.json`**:
+   ```json
+   {
+     "coverage-gutters.coverageBaseDir": "${workspaceFolder}",
+     "coverage-gutters.coverageFileNames": ["coverage.xml"],
+     "coverage-gutters.showLineCoverage": true,
+     "coverage-gutters.showRulerCoverage": true
+   }
+   ```
+3. **Run tests to generate coverage**:
+   ```bash
+   pytest tests/ --cov=model_manager --cov-report=xml
+   ```
+4. **Activate Coverage Gutters**: Press `Ctrl+Shift+7` (or `Cmd+Shift+7` on Mac)
+
+#### Automated Versioning
+
+The project uses semantic versioning based on branch patterns:
+
+| Branch Pattern | Version Change | Example |
+|---------------|----------------|---------|
+| `release/*` | Major version bump | `2.0.0` |
+| `feature/*` | Minor version bump | `1.2.0` |
+| `fix/*` | Patch version bump | `1.1.3` |
+| `rc/*` | Release candidate | `1.1.2-rc2` |
+
+Version is calculated automatically in the CI/CD pipeline and passed to SonarQube.
 
 ### Project Structure
 ```
 model_manager/
 ├── activities/              # Temporal activity implementations
-│   ├── activities.py       # Main activities orchestrator
+│   ├── __init__.py
-│   ├── gates.py            # Data quality gates and filtering
+│   ├── activities.py       # Main activities orchestrator (combines all activities)
-│   └── mlflow.py           # MLFlow model operations
+│   ├── gates.py            # Data quality gates and filtering logic
+│   ├── minio.py            # MinIO object storage operations
+│   └── mlflow.py           # MLFlow model operations (predict/transform)
 ├── workflows/               # Temporal workflow definitions
-│   ├── predictions_batch.py # Main batch prediction workflow
+│   ├── __init__.py
+│   ├── predictions_batch.py # Main batch prediction workflow entry point
 │   ├── minimal_retrain.py  # Model retraining workflow
 │   └── sub_workflows/      # Sub-workflow implementations
-│       ├── prediction_process.py # Core prediction workflow
+│       ├── __init__.py
-│       └── format_and_export_prediction.py # Export workflow
+│       ├── prediction_process.py # Core prediction pipeline
+│       └── format_and_export_prediction.py # Data export workflow
 ├── worker/                  # Worker implementation
-│   └── worker.py           # Main worker orchestrator
+│   ├── __init__.py
-├── utils/                   # Utility functions
+│   └── worker.py           # Main worker orchestrator (Temporal client setup)
-│   ├── connectors_config.py # Database configuration
+├── utils/                   # Utility functions and helpers
-│   ├── filters/            # Data quality filters
+│   ├── __init__.py
-│   │   ├── conditional_filters.py # Conditional data filters
+│   ├── connectors_config.py # Environment-based configuration builders
-│   │   └── mlflow_filters.py # MLFlow response filters
+│   ├── filters/            # Data quality validation filters
+│   │   ├── __init__.py
+│   │   ├── conditional_filters.py # Input data validation filters
+│   │   └── mlflow_filters.py # MLFlow response validation filters
 │   └── repository/         # Data access layer
-│       └── model_repository.py # MLFlow model operations
+│       ├── __init__.py
+│       └── model_repository.py # MLFlow model operations and retraining
 ├── metrics.py               # Prometheus metrics definitions
 └── __init__.py
 ```
@@ -889,10 +1075,28 @@ model_manager/
 1. **Follow Temporal patterns** for new workflows and activities
 2. **Add comprehensive docstrings** for all public methods
 3. **Include Prometheus metrics** for monitoring
-4. **Add unit tests** for new functionality
+4. **Add unit tests** for new functionality (maintain 80%+ coverage)
-5. **Update this README** with new features and configuration
+5. **Run validation script** (`./validate.sh`) before committing
+6. **Update this README** with new features and configuration
 
-## 🐛 Troubleshooting
+### Test Coverage Guidelines
+
+- **Minimum coverage**: 80% (enforced by CI/CD)
+- **Current coverage**: 99%+ 🎯
+- **Test all branches**: Use Coverage Gutters to identify uncovered lines
+- **Mock external dependencies**: Use `unittest.mock` for external services
+- **Async testing**: Use `pytest-asyncio` for async activities and workflows
+- **Test structure**:
+  ```
+  tests/
+  ├── laborious/
+  │   ├── activities/      # Activity tests
+  │   ├── workflows/       # Workflow tests
+  │   ├── utils/          # Utility tests
+  │   └── worker/         # Worker tests
+  ```
+
+## Troubleshooting
 
 ### Common Issues
 
@@ -923,7 +1127,7 @@ Enable debug logging by setting the log level:
 export LOG_LEVEL=DEBUG
 ```
 
-## ⚡ Performance Tuning
+## Performance Tuning
 
 ### Key Parameters
 
@@ -939,7 +1143,7 @@ export LOG_LEVEL=DEBUG
 - **Database Performance**: Optimize indexes and connection pooling
 - **MLFlow Performance**: Configure appropriate model serving resources
 
-## 🤝 Contributing
+## Contributing
 
 1. Fork the repository
 2. Create a feature branch
@@ -955,11 +1159,11 @@ export LOG_LEVEL=DEBUG
 - Use type hints where appropriate
 - Follow Temporal.io best practices
 
-## 📄 License
+## License
 
 This project is licensed under the terms specified in the LICENSE file.
 
-## 🆘 Support
+## Support
 
 For support and questions:
 - Check the troubleshooting section above