SIENTIAPDE-1243: Remove OPC server integration and add code quality tools.

This commit removes the OPC server integration from the Model Manager, including related activities, repositories, metrics, and configuration. It also adds code quality tools such as Ruff (linting/formatting), mypy (type checking), and Bandit (security analysis) along with a validation script and CI/CD integration for automated code validation. The README has been updated to reflect these changes.
2025-10-01 16:25:08 -03:00
parent bc4d98f78d
commit b102f79087
24 changed files with 453 additions and 1810 deletions
--- a/README.md
+++ b/README.md
@@ -9,7 +9,7 @@ A comprehensive AI model management platform for the complete machine learning l
 - **Temporal Workflow Orchestration**: Robust workflow management with automatic retry policies and fault tolerance
 - **Data Quality Gates**: Configurable filtering for data validation, MLFlow API responses, and custom validation rules
 - **Multi-Model Support**: Flexible ML model management with retention policies and versioning
- **Real-time Data Export**: PostgreSQL persistence and OPC server integration for industrial systems
+- **Real-time Data Export**: PostgreSQL persistence for data storage
 - **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility

 ### Advanced Capabilities
@@ -19,6 +19,12 @@ A comprehensive AI model management platform for the complete machine learning l
 - **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support
 - **Model Retraining**: Automated model retraining workflows with production model updates

+### Development & Quality Assurance
+- **Code Quality Tools**: Ruff (linting/formatting), mypy (type checking), Bandit (security analysis)
+- **Automated Validation**: Pre-commit validation script and CI/CD integration
+- **Comprehensive Testing**: pytest with async support and 70%+ code coverage
+- **Type Safety**: Static type checking with mypy for improved code reliability
+
 ## Architecture

 The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
@@ -60,7 +66,6 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
  - Task queue configuration and load balancing
  - Prometheus metrics server initialization
  - Notification handler setup and configuration
-  - OPC server connection management
 - **Key Features**:
  - Automatic scaling with `PollerBehaviorAutoscaling`
  - Health check endpoints for Kubernetes liveness/readiness probes
@@ -83,20 +88,16 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
 - **Activities**: Main activity orchestrator combining all functionality through multiple inheritance
 - **Gates**: Data quality validation and filtering mechanisms
 - **MLFlow**: Model transformation and prediction operations
- **OPC**: Real-time data export to industrial OPC servers
 - **Key Features**:
  - Multiple inheritance pattern for unified activity interface
  - Configurable filter policies and validation rules
  - MLFlow model serving integration with configurable flavors
-  - OPC UA client with certificate-based authentication
  - Comprehensive error handling and notification integration
-  - Support for multiple OPC servers with independent configurations

 #### **Data Services (`model_manager/utils/`)**
 - **Connectors Config**: Environment variable-based configuration management
- **Repository**: Data access layer for MLFlow and OPC operations
+- **Repository**: Data access layer for MLFlow operations
  - `model_repository.py`: MLFlow model operations and retraining
-  - `opc_repository.py`: OPC server communication and data writing
 - **Filters**: Data quality validation and MLFlow response filtering
  - `conditional_filters.py`: Input data validation filters
  - `mlflow_filters.py`: MLFlow API response validation filters
@@ -105,14 +106,14 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
  - Connection pool management and optimization
  - Security credential management
  - Configuration validation and error handling
-  - Support for multiple OPC servers and MLFlow model flavors
+  - Support for MLFlow model flavors

 ### Data Flow Architecture

 #### **1. Batch Prediction Pipeline**
 ```
 Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform → 
-MLFlow Prediction → Response Validation → Export (PostgreSQL + OPC)
+MLFlow Prediction → Response Validation → Export (PostgreSQL)
 ```

 #### **2. Model Retraining Pipeline**
@@ -121,16 +122,10 @@ Training Data → Model Retraining → Quality Validation →
 Production Update → Notification & Monitoring
 ```

-#### **3. Real-time Export Pipeline**
-```
-Prediction Results → Data Formatting → OPC Server Write → 
-Success/Failure Metrics → Notification System
-```
-
 ### Security Architecture

 #### **Authentication & Authorization**
- **Certificate-based OPC Authentication**: Secure industrial communication
+
 - **MLFlow API Authentication**: Username/password with secure transmission
 - **Database Connection Security**: Encrypted connections with credential management
 - **Kubernetes Secrets Integration**: Secure credential storage and access
@@ -195,11 +190,7 @@ The **PredictionsBatch** workflow is the main entry point for batch prediction p
    "API_ERROR": {"POLICY": "STOP"}
  },
  "model_retention": 60,
-  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
-  "opc_output_config": {
-    "server_id": "opc_server_1",
-    "tags": ["prediction_output"]
-  }
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"]
 }
 ```

@@ -267,8 +258,7 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M
    "NAN_VALUES": {"POLICY": "STOP"}
  },
  "model_retention": 60,
-  "path_priority": ["STOP", "CONTINUE", "REPEAT"],
-  "opc_output_config": {...}
+  "path_priority": ["STOP", "CONTINUE", "REPEAT"]
 }
 ```

@@ -288,33 +278,30 @@ flowchart LR
 The **FormatAndExportPrediction** workflow handles prediction data formatting and export operations to multiple destinations.

 #### Purpose
- **Data Formatting**: Formats prediction data for different output destinations
+- **Data Formatting**: Formats prediction data for database storage
 - **PostgreSQL Export**: Persists predictions to database with metrics
- **OPC Integration**: Writes predictions to OPC servers for real-time access
 - **Metrics Recording**: Tracks export operations and performance metrics

 #### Execution Flow
 1. **Path Decision**: Determines formatting path based on configuration
 2. **Data Formatting**: Formats prediction data for specific output requirements
 3. **PostgreSQL Export**: Writes formatted predictions to database
-4. **OPC Export**: Writes predictions to OPC servers
-5. **Metrics Recording**: Records export performance and success metrics
+4. **Metrics Recording**: Records export performance and success metrics

 #### Key Features
 - **Flexible Formatting**: Configurable output formats for different destinations
- **Multi-Destination Export**: PostgreSQL and OPC server integration
+- **Database Export**: PostgreSQL integration for data persistence
 - **Performance Monitoring**: Comprehensive metrics for export operations
 - **Error Handling**: Robust error handling with notification integration

 #### Architecture Diagram
 ```mermaid
 flowchart LR
-    A[1. format_prediction/format_default_prediction] --> B[2. write_opc_data] --> C[3. export_data_to_postgres] --> D[4. write_metrics]
+    A[1. format_prediction/format_default_prediction] --> B[2. export_data_to_postgres] --> C[3. write_metrics]
    
    A -.-> Format[Data Formatting]
-    B -.-> OPC[OPC Servers]
-    C -.-> PostgreSQL[(PostgreSQL)]
-    D -.-> Prometheus[Prometheus]
+    B -.-> PostgreSQL[(PostgreSQL)]
+    C -.-> Prometheus[Prometheus]
 ```

 ### 4. Minimal Retrain Workflow (`minimal_retrain.py`)
@@ -351,7 +338,6 @@ flowchart LR
 - Temporal server/cluster
 - PostgreSQL database
 - MLFlow server
- OPC server(s)
 - MongoDB server (for notifications)

 **Note**: External dependencies must be available either through:
@@ -398,12 +384,10 @@ flowchart LR
  4. **Install Python dependencies**
    ```bash
    python -m pip install --upgrade pip
+    # Install production dependencies
    pip install -r requirements.txt
-    ```
-
-  5. **Install test libraries**
-    ```bash
-    pip install pytest pytest-cov pytest-asyncio
+    # Install development and testing tools
+    pip install -r requirements-dev.txt
    ```

 4. **Create environment configuration file**
@@ -499,6 +483,113 @@ fi
 python -m model_manager.worker.worker
 ```

+## 🔍 Code Quality & Validation
+
+### Overview
+
+Como Python não é uma linguagem compilada, utilizamos um conjunto robusto de ferramentas para validar a qualidade, segurança e correção do código antes da execução. Estas ferramentas detectam erros, problemas de estilo, vulnerabilidades de segurança e garantem a consistência do código.
+
+### Ferramentas de Validação
+
+#### 1. **Ruff** - Linting e Formatação ⚡
+Ferramenta moderna e extremamente rápida (escrita em Rust) que substitui múltiplas ferramentas:
+- **Linting**: Detecta erros de código, problemas de estilo (PEP 8), bugs comuns
+- **Formatação**: Formata código automaticamente de forma consistente
+- **Velocidade**: 10-100x mais rápido que Flake8/Black
+
+#### 2. **mypy** - Type Checking 🏷️
+Verificador de tipos estáticos que analisa type hints:
+- Detecta erros de tipo antes da execução
+- Melhora a documentação do código
+- Previne bugs relacionados a tipos incorretos
+
+#### 3. **Bandit** - Análise de Segurança 🔒
+Scanner de vulnerabilidades de segurança:
+- Detecta padrões inseguros de código
+- Identifica hardcoded passwords, SQL injection, etc.
+- Garante conformidade com práticas de segurança
+
+#### 4. **pytest** - Testes Automatizados 🧪
+Framework de testes com cobertura de código:
+- Executa testes unitários e de integração
+- Mede cobertura de código
+- Suporta testes assíncronos
+
+### Instalação das Ferramentas
+
+```bash
+# Instalar dependências de desenvolvimento
+pip install -r requirements-dev.txt
+```
+
+### Validação Completa
+
+#### Opção 1: Script Automatizado (Recomendado)
+```bash
+# Executar todas as validações de uma vez
+./validate.sh
+```
+
+O script `validate.sh` executa automaticamente:
+1. ✅ Verificação de formatação (Ruff)
+2. ✅ Linting de código (Ruff)
+3. ✅ Type checking (mypy)
+4. ✅ Análise de segurança (Bandit)
+5. ✅ Testes unitários com cobertura (pytest)
+
+#### Opção 2: Comandos Individuais
+```bash
+# 1. Verificar formatação
+ruff format --check model_manager/ tests/
+
+# 2. Verificar linting
+ruff check model_manager/ tests/
+
+# 3. Verificar tipos
+mypy model_manager/
+
+# 4. Análise de segurança
+bandit -r model_manager/ -ll
+
+# 5. Executar testes
+pytest tests/ --cov=model_manager --cov-report=term-missing
+```
+
+### Correção Automática
+
+Algumas ferramentas podem corrigir problemas automaticamente:
+
+```bash
+# Formatar código automaticamente
+ruff format model_manager/ tests/
+
+# Corrigir problemas de linting automaticamente
+ruff check --fix model_manager/ tests/
+```
+
+### Configuração
+
+Todas as ferramentas são configuradas no arquivo `pyproject.toml`:
+- **Ruff**: Regras de linting, formatação, complexidade
+- **mypy**: Configurações de type checking
+- **pytest**: Opções de teste e cobertura
+- **Bandit**: Regras de segurança
+
+### Integração com CI/CD
+
+O workflow `.github/workflows/quality-gate.yml` executa automaticamente todas as validações em cada push/PR:
+- ✅ Formatação e linting bloqueiam merge se falharem
+- ⚠️ Type checking e segurança geram avisos mas não bloqueiam
+- ✅ Testes devem passar com cobertura mínima de 70%
+
+### Boas Práticas
+
+1. **Antes de Commit**: Execute `./validate.sh` para garantir qualidade
+2. **Durante Desenvolvimento**: Use `ruff check --watch` para feedback em tempo real
+3. **Type Hints**: Adicione type hints em funções novas para melhor validação
+4. **Testes**: Mantenha cobertura acima de 70%
+5. **Segurança**: Revise e corrija todos os avisos do Bandit
+
 ## 🧪 Testing

 ### Test Structure
@@ -540,13 +631,6 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
  - Labels: `pod_id`, `model_name`, `pipeline_name`
  - Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]

-### OPC Export Metrics
- `model_manager_prediction_opc_writing_count`: Counter for OPC server write operations
-  - Labels: `pod_id`, `model_name`, `pipeline_name`, `opc_server_id`
- `model_manager_prediction_opc_writing_response_time_monitor`: Histogram for OPC write response times
-  - Labels: `pod_id`, `model_name`, `pipeline_name`, `opc_server_id`
-  - Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]
-
 ### Data Quality Metrics
 - Filter pass/fail rates through notification system
 - MLFlow API response validation metrics
@@ -571,14 +655,6 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 | `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
 | `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
 | `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
-| `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
-| `OPC_ID` | OPC server identifier | `1` | No |
-| `OPC_URL` | OPC server URL | `opc.tcp://localhost:4840` | No |
-| `OPC_SERVER_URI` | OPC server URI | `opc.tcp://localhost:4840` | No |
-| `OPC_CERT_PATH` | OPC client certificate path | `None` | No |
-| `OPC_PRIVATE_KEY_PATH` | OPC private key path | `None` | No |
-| `OPC_SERVER_CERT_PATH` | OPC server certificate path | `None` | No |
-| `OPC_RECONNECTION_INTERVAL` | OPC reconnection interval (ms) | `120` | No |
 | `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
 | `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
 | `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
@@ -590,45 +666,6 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
 | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
 | `POD_ID` | Kubernetes pod identifier | `None` | No |

-
-
-
-### OPC Configuration
-
-For multiple OPC servers, use the `OPC_CONFIG` environment variable:
-
-```json
-{
-  "opc_server_1": {
-    "url": "opc.tcp://server1:4840",
-    "name": "Server1",
-    "server_uri": "urn:server1:opcua",
-    "cert_path": "/path/to/cert.pem",
-    "private_key_path": "/path/to/key.pem",
-    "server_cert_path": "/path/to/server_cert.pem",
-    "reconnection_interval": 5000
-  },
-  "opc_server_2": {
-    "url": "opc.tcp://server2:4840",
-    "name": "Server2",
-    "server_uri": "urn:server2:opcua",
-    "cert_path": "/path/to/cert.pem",
-    "private_key_path": "/path/to/key.pem",
-    "server_cert_path": "/path/to/server_cert.pem",
-    "reconnection_interval": 5000
-  }
-}
-```
-
-For single OPC server, use individual environment variables:
- `OPC_URL`
- `OPC_NAME`
- `OPC_SERVER_URI`
- `OPC_CERT_PATH`
- `OPC_PRIVATE_KEY_PATH`
- `OPC_SERVER_CERT_PATH`
- `OPC_RECONNECTION_INTERVAL`
-
 ### Workflow Configuration

 MongoDB pipeline configuration:
@@ -705,7 +742,6 @@ This is the configuration created by the Orchestrator in Temporal.
  },
  "model_id":"352",
  "model_name":"courier",
-  "opc_output_config":{},
  "path_priority":["STOP","CONTINUE","REPEAT"],
  "predictions_storage_policy":"lts:1",
  "query":"select * from sientia_data.laborious_data where model_id = 352 order by \"timestamp\" desc limit 300;",
@@ -726,8 +762,7 @@ model_manager/
 ├── activities/              # Temporal activity implementations
 │   ├── activities.py       # Main activities orchestrator
 │   ├── gates.py            # Data quality gates and filtering
-│   ├── mlflow.py           # MLFlow model operations
-│   └── opc.py              # OPC server operations
+│   └── mlflow.py           # MLFlow model operations
 ├── workflows/               # Temporal workflow definitions
 │   ├── predictions_batch.py # Main batch prediction workflow
 │   ├── minimal_retrain.py  # Model retraining workflow
@@ -742,8 +777,7 @@ model_manager/
 │   │   ├── conditional_filters.py # Conditional data filters
 │   │   └── mlflow_filters.py # MLFlow response filters
 │   └── repository/         # Data access layer
-│       ├── model_repository.py # MLFlow model operations
-│       └── opc_repository.py # OPC server operations
+│       └── model_repository.py # MLFlow model operations
 ├── metrics.py               # Prometheus metrics definitions
 └── __init__.py
 ```
@@ -775,12 +809,7 @@ model_manager/
   - Check connection credentials and network access
   - Ensure proper connection pool configuration

-4. **OPC Connection Failures**
-   - Verify OPC server is accessible
-   - Check certificate and key file paths
-   - Review OPC server logs for connection issues
-
-5. **Workflow Execution Failures**
+4. **Workflow Execution Failures**
   - Review activity error logs and notifications
   - Check data quality filter configurations
   - Verify input data format and required fields