SIENTIAPDE-1231

Enhance README and repository utilities for clarity and functionality - Updated README.md to improve descriptions and structure, adding detailed sections for features, workflows, and architecture. - Enhanced MinioRepository with comprehensive docstrings for methods and class attributes, improving usability and documentation. - Refined MLFlowRepository with clearer method descriptions and improved logging for better observability and maintainability.
2025-10-16 11:01:36 -03:00
parent de47820c4a
commit 644a43093a
3 changed files with 291 additions and 190 deletions
--- a/README.md
+++ b/README.md
@@ -1,42 +1,108 @@
 # Sientia DataOps Laborious

-A high-performance, scalable machine learning prediction system built on Temporal.io for industrial data processing and ML model inference. The Laborious system provides enterprise-grade ML model management, batch prediction processing, and real-time data export capabilities with comprehensive data quality validation and monitoring.
+A comprehensive, Temporal-based ML orchestration system for industrial data processing and model inference. Laborious delivers enterprise-grade batch prediction, model management, optional real-time export (OPC), and automated retraining with strong data quality validation and observability.
+
+## 📑 Table of Contents
+
+- [Features](#features)
+  - [Core Functionality](#core-functionality)
+  - [Advanced Capabilities](#advanced-capabilities)
+  - [Development & Quality Assurance](#development--quality-assurance)
+- [Architecture](#architecture)
+  - [Architecture Principles](#architecture-principles)
+  - [Key Components](#key-components)
+  - [Data Flow Architecture](#data-flow-architecture)
+  - [Security Architecture](#security-architecture)
+- [Workflows](#workflows)
+  - [Predictions Batch Workflow](#1-predictions-batch-workflow-predictions_batchpy)
+  - [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
+  - [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
+  - [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
+- [Installation & Setup](#installation--setup)
+  - [Prerequisites](#prerequisites)
+  - [Environment Setup](#environment-setup)
+  - [Temporal Namespace Setup](#temporal-namespace-setup)
+  - [Local Development Setup](#local-development-setup)
+- [How to Run](#how-to-run)
+  - [Running the Laborious Application](#running-the-laborious-application)
+  - [Running Tests and Coverage](#running-tests-and-coverage)
+  - [Manual Test Execution](#manual-test-execution)
+  - [Manual Application Execution](#manual-application-execution)
+- [Code Quality & Validation](#code-quality--validation)
+  - [Overview](#overview)
+  - [Validation Tools](#validation-tools)
+  - [Tools Installation](#tools-installation)
+  - [Complete Validation](#complete-validation)
+  - [Automatic Fixes](#automatic-fixes)
+  - [Configuration](#configuration)
+  - [CI/CD Integration](#cicd-integration)
+  - [Best Practices](#best-practices)
+- [Testing](#testing)
+  - [Test Structure](#test-structure)
+  - [Test Execution](#test-execution)
+- [Monitoring and Metrics](#monitoring-and-metrics)
+  - [Application Health Metrics](#application-health-metrics)
+  - [Prediction Operation Metrics](#prediction-operation-metrics)
+  - [OPC Export Metrics](#opc-export-metrics)
+  - [Data Quality Metrics](#data-quality-metrics)
+- [Configuration](#configuration-1)
+  - [Environment Variables](#environment-variables)
+  - [OPC Configuration](#opc-configuration)
+  - [Workflow Configuration](#workflow-configuration)
+- [Development](#development)
+  - [Project Structure](#project-structure)
+  - [Adding New Features](#adding-new-features)
+- [Troubleshooting](#troubleshooting)
+  - [Common Issues](#common-issues)
+  - [Debug Mode](#debug-mode)
+- [Performance Tuning](#performance-tuning)
+  - [Key Parameters](#key-parameters)
+  - [Scaling Considerations](#scaling-considerations)
+- [Contributing](#contributing)
+  - [Code Quality Standards](#code-quality-standards)
+- [License](#license)
+- [Support](#support)

 ## Features

 ### Core Functionality
- **Batch Prediction Processing**: High-throughput ML model inference using MLFlow models
- **Temporal Workflow Orchestration**: Robust workflow management with automatic retry policies and fault tolerance
- **Data Quality Gates**: Configurable filtering for data validation, MLFlow API responses, and custom validation rules
- **Multi-Model Support**: Flexible ML model management with retention policies and versioning
- **Real-time Data Export**: PostgreSQL persistence and OPC server integration for industrial systems
- **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility
+- **Batch Prediction Processing**: High-throughput ML inference using MLFlow models
+- **Temporal Workflow Orchestration**: Robust workflow management with retries and fault tolerance
+- **Data Quality Gates**: Configurable filtering for input data and MLFlow API responses
+- **Multi-Model Support**: Flexible model management with retention and versioning
+- **Optional Real-time Export**: PostgreSQL persistence and OPC server integration for industrial systems
+- **Comprehensive Monitoring**: Prometheus metrics and structured logging for observability

 ### Advanced Capabilities
- **Incremental Data Processing**: Timestamp-based data loading to avoid reprocessing
+- **Incremental Data Processing**: Timestamp-based loading to avoid reprocessing
 - **Configurable Data Retention**: Model retention policies with automatic cleanup
- **Notification System**: Integrated alerting and notification management via MongoDB
- **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support
- **Model Retraining**: Automated model retraining workflows with production model updates
+- **Notification System**: Integrated alerting via MongoDB
+- **Scalable Architecture**: Kubernetes-ready with horizontal scaling
+- **Model Retraining**: Automated retraining workflows with production model updates
+
+### Development & Quality Assurance
+- **Code Quality Tools**: Ruff (lint/format), mypy (types), Bandit (security)
+- **Automated Validation**: `validate.sh` and CI quality gates
+- **Comprehensive Testing**: pytest with async support and high coverage
+- **Type Safety**: Static type checking with mypy
+- **Coverage Visualization**: Coverage Gutters integration

 ## Architecture

-The Laborious system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
-
+Laborious uses a Temporal-based architecture with strong separation of concerns and defensive error handling for production ML.

 ### Architecture Principles

 #### 1. **Separation of Concerns**
- **Worker Layer**: Manages Temporal workers, task queues, and application lifecycle
- **Workflow Layer**: Orchestrates business logic and process coordination
- **Activity Layer**: Implements specific operations and external system interactions
- **Data Layer**: Handles data persistence, caching, and external service connections
+- **Worker Layer**: Temporal workers, task queues, lifecycle
+- **Workflow Layer**: Business orchestration and coordination
+- **Activity Layer**: External system interactions and isolated operations
+- **Data Layer**: Persistence, caching, connectors

 #### 2. **Fault Tolerance & Resilience**
- **Automatic Retry Policies**: Configurable retry strategies for transient failures
- **Circuit Breaker Pattern**: Prevents cascading failures in external service calls
- **Graceful Degradation**: System continues operating with reduced functionality
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
+- **Automatic Retry Policies** for transient failures
+- **Graceful Degradation** and circuit breaking for dependencies
+- **Detailed Error Handling** with notifications

 #### 3. **Scalability & Performance**
 - **Horizontal Scaling**: Multiple worker instances for load distribution
@@ -53,101 +119,57 @@ The Laborious system uses a Temporal-based workflow architecture with clear sepa
 ### Key Components

 #### **Worker (`laborious/worker/worker.py`)**
- **Purpose**: Main application orchestrator managing Temporal workers and task queues
- **Responsibilities**:
-  - Temporal client initialization and connection management
-  - Worker lifecycle management and graceful shutdown
-  - Task queue configuration and load balancing
-  - Prometheus metrics server initialization
-  - Notification handler setup and configuration
-  - OPC server connection management
- **Key Features**:
-  - Automatic scaling with `PollerBehaviorAutoscaling`
-  - Health check endpoints for Kubernetes liveness/readiness probes
-  - Graceful shutdown with cleanup procedures
-  - Multi-instance deployment support
-  - Two dedicated task queues: `predictions_batch-queue` and `minimal_retrain-queue`
+- Temporal client setup, worker lifecycle, task queues
+- Metrics server initialization, notification handler setup
+- Graceful shutdown and autoscaling-friendly behavior

 #### **Workflows (`laborious/workflows/`)**
- **PredictionsBatch**: Main entry point for batch prediction pipelines
- **PredictionProcess**: Core prediction pipeline with MLFlow integration
- **FormatAndExportPrediction**: Data formatting and export operations
- **MinimalRetrain**: Automated model retraining and deployment
- **Key Features**:
-  - Temporal workflow definitions with retry policies
-  - Child workflow orchestration and delegation
-  - Comprehensive error handling and recovery
-  - Configurable timeout and retry strategies
+- `predictions_batch.py`: Batch prediction entry point
+- `sub_workflows/prediction_process.py`: Core prediction pipeline
+- `sub_workflows/format_and_export_prediction.py`: Formatting and export
+- `minimal_retrain.py`: Automated model retraining and production update

 #### **Activities (`laborious/activities/`)**
- **Activities**: Main activity orchestrator combining all functionality through multiple inheritance
- **Gates**: Data quality validation and filtering mechanisms
- **MLFlow**: Model transformation and prediction operations
- **OPC**: Real-time data export to industrial OPC servers
- **Key Features**:
-  - Multiple inheritance pattern for unified activity interface
-  - Configurable filter policies and validation rules
-  - MLFlow model serving integration with configurable flavors
-  - OPC UA client with certificate-based authentication
-  - Comprehensive error handling and notification integration
-  - Support for multiple OPC servers with independent configurations
+- `gates.py`: Data quality validation and filtering
+- `mlflow.py`: Transform and predict operations
+- `opc.py`: OPC UA export to industrial systems (optional)
+- `activities.py`: Aggregates activity interfaces

 #### **Data Services (`laborious/utils/`)**
- **Connectors Config**: Environment variable-based configuration management
- **Repository**: Data access layer for MLFlow and OPC operations
-  - `model_repository.py`: MLFlow model operations and retraining
-  - `opc_repository.py`: OPC server communication and data writing
- **Filters**: Data quality validation and MLFlow response filtering
-  - `conditional_filters.py`: Input data validation filters
-  - `mlflow_filters.py`: MLFlow API response validation filters
- **Key Features**:
-  - Environment variable-based configuration with sensible defaults
-  - Connection pool management and optimization
-  - Security credential management
-  - Configuration validation and error handling
-  - Support for multiple OPC servers and MLFlow model flavors
+- `connectors_config.py`: Env-driven configuration builders
+- `repository/model_repository.py`: MLFlow operations and retraining
+- `repository/opc_repository.py`: OPC communication and writes
+- `filters/conditional_filters.py` and `filters/mlflow_filters.py`

 ### Data Flow Architecture

 #### **1. Batch Prediction Pipeline**
 ```
-Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform → 
-MLFlow Prediction → Response Validation → Export (PostgreSQL + OPC)
+Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform →
+MLFlow Prediction → Response Validation → Export (PostgreSQL [+ OPC])
 ```

 #### **2. Model Retraining Pipeline**
 ```
-Training Data → Model Retraining → Quality Validation → 
+Training Data → Model Retraining → Quality Validation →
 Production Update → Notification & Monitoring
 ```

-#### **3. Real-time Export Pipeline**
-```
-Prediction Results → Data Formatting → OPC Server Write → 
-Success/Failure Metrics → Notification System
-```
-
 ### Security Architecture

 #### **Authentication & Authorization**
- **Certificate-based OPC Authentication**: Secure industrial communication
- **MLFlow API Authentication**: Username/password with secure transmission
- **Database Connection Security**: Encrypted connections with credential management
- **Kubernetes Secrets Integration**: Secure credential storage and access
+- **MLFlow API Authentication**: Username/password
+- **Database Security**: Encrypted connections and credential management
+- **OPC Certificates** (if enabled): Client/server certs
+- **Kubernetes Secrets**: Secure secret storage

 #### **Network Security**
- **TLS/SSL Encryption**: Secure communication channels
- **Network Isolation**: Kubernetes network policies and service mesh
- **Firewall Rules**: Controlled access to external services
- **VPN Integration**: Secure remote access and management
+- TLS/SSL, network policies, service mesh, firewalls, VPN

 #### **Data Security**
- **Data Encryption**: At-rest and in-transit encryption
- **Access Control**: Role-based access control (RBAC)
- **Audit Logging**: Comprehensive access and operation logging
- **Data Retention**: Configurable data lifecycle management
+- At-rest/in-transit encryption, RBAC, audit logging, lifecycle management

-## 🔄 Workflows
+## Workflows

 ### 1. Predictions Batch Workflow (`predictions_batch.py`)

@@ -351,8 +373,9 @@ flowchart LR
 - Temporal server/cluster
 - PostgreSQL database
 - MLFlow server
- OPC server(s)
+- MinIO object storage (for MLFlow artifacts)
 - MongoDB server (for notifications)
+- OPC server(s) if using OPC export

 **Note**: External dependencies must be available either through:
 - Kubernetes cluster deployment