gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1182

Browse Source 
Update README.md to enhance architecture documentation

- Removed detailed system overview diagram to streamline content.
- Added architecture diagrams for key workflows: PredictionsBatch, PredictionProcess, FormatAndExportPrediction, and MinimalRetrain.
- Improved clarity and structure of the architecture principles section.
This commit is contained in:
vitor-aignosi
2025-08-29 16:59:00 -03:00
parent d37a53f5ea
commit f9784b8f3e
1 changed files with 41 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files

130
README.md
View File

@@ -23,95 +23,6 @@ A high-performance, scalable machine learning prediction system built on Tempora
The Laborious system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
### System Overview
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ Temporal Cluster │
│ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │
│ │ Main Worker │ │ Temporal Client │ │ Task Queues │ │
│ │ │◄──►│ │◄──►│ │ │
│ │ - Metrics Server│ │ - Namespace Mgmt │ │ - predictions_batch-queue│ │
│ │ - Notifications │ │ - Runtime Config │ │ - minimal_retrain-queue │ │
│ │ - Lifecycle │ │ - Connection │ │ - Auto-scaling │ │
│ │ - Health Checks │ │ - Security │ │ - Load Balancing │ │
│ └─────────────────┘ └──────────────────┘ └─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Workflow Layer │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ PredictionsBatch│ │ Sub-Workflows │ │
│ │ │ │ │ │
│ │ - Data Loading │ │ - PredictionProcess │ │
│ │ - Configuration │ │ - FormatAndExportPrediction │ │
│ │ - Delegation │ │ - Error Handling │ │
│ └─────────────────┘ └─────────────────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ MinimalRetrain │ │ Model Management │ │
│ │ │ │ │ │
│ │ - Retraining │ │ - Version Control │ │
│ │ - Validation │ │ - Production Updates │ │
│ │ - Deployment │ │ - Quality Assurance │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Activity Layer │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ Data Quality │ │ MLFlow Operations │ │
│ │ │ │ │ │
│ │ - Input Gates │ │ - Model Transform │ │
│ │ - Validation │ │ - Model Prediction │ │
│ │ - Filtering │ │ - Response Validation │ │
│ │ - Policy Mgmt │ │ - Error Handling │ │
│ └─────────────────┘ └─────────────────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ Storage Ops │ │ OPC Operations │ │
│ │ │ │ │ │
│ │ - PostgreSQL │ │ - Server Connections │ │
│ │ - Data Export │ │ - Tag Writing │ │
│ │ - Metrics │ │ - Real-time Export │ │
│ │ - Cleanup │ │ - Error Recovery │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Data Services │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ PostgreSQL │ │ MongoDB │ │
│ │ │ │ │ │
│ │ - Predictions │ │ - Notifications │ │
│ │ - Metadata │ │ - Audit Logs │ │
│ │ - Metrics │ │ - Configuration │ │
│ │ - Cleanup │ │ - User Management │ │
│ └─────────────────┘ └─────────────────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ MLFlow API │ │ OPC Servers │ │
│ │ │ │ │ │
│ │ - Model Serving │ │ - Real-time Data │ │
│ │ - Transform │ │ - Industrial Integration │ │
│ │ - Prediction │ │ - Security & Auth │ │
│ │ - Versioning │ │ - Load Balancing │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ External Systems │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ Prometheus │ │ Kubernetes │ │
│ │ │ │ │ │
│ │ - Metrics │ │ - Orchestration │ │
│ │ - Alerting │ │ - Scaling │ │
│ │ - Dashboards │ │ - Health Checks │ │
│ │ - Monitoring │ │ - Resource Management │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
### Architecture Principles
@@ -284,6 +195,14 @@ The **PredictionsBatch** workflow is the main entry point for batch prediction p
}
```
#### Architecture Diagram
```mermaid
flowchart LR
A[1. load_custom_query] --> B[2. prediction_process 🔃]
A -.-> Database[(Database)]
```
### 2. Prediction Process Workflow (`prediction_process.py`)
The **PredictionProcess** workflow implements the core prediction pipeline for ML model inference. It handles data quality validation, MLFlow model interactions, and prediction processing.
@@ -345,6 +264,17 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M
}
```
#### Architecture Diagram
```mermaid
flowchart LR
A[1. get_last_timestamp] --> B[2. input_gate] --> C[3. request_transform] --> D[4. mlflow_response_gate] --> E[5. mlflow_content_gate] --> F[6. request_predict] --> G[7. mlflow_response_gate] --> H[8. format_and_export_prediction🔃]
A -.-> Redis[(Redis)]
C -.-> MLFlow[MLFlow]
F -.-> MLFlow[MLFlow]
G -.-> Filters[MLFlow Filters]
```
### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)
The **FormatAndExportPrediction** workflow handles prediction data formatting and export operations to multiple destinations.
@@ -368,6 +298,17 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an
- **Performance Monitoring**: Comprehensive metrics for export operations
- **Error Handling**: Robust error handling with notification integration
#### Architecture Diagram
```mermaid
flowchart LR
A[1. format_prediction/format_default_prediction] --> B[2. write_opc_data] --> C[3. export_data_to_postgres] --> D[4. write_metrics]
A -.-> Format[Data Formatting]
B -.-> OPC[OPC Servers]
C -.-> PostgreSQL[(PostgreSQL)]
D -.-> Prometheus[Prometheus]
```
### 4. Minimal Retrain Workflow (`minimal_retrain.py`)
The **MinimalRetrain** workflow handles automated model retraining and production model updates.
@@ -385,6 +326,17 @@ The **MinimalRetrain** workflow handles automated model retraining and productio
4. **Production Update**: Updates production model if quality criteria met
5. **Data Export**: Exports training data for analysis
#### Architecture Diagram
```mermaid
flowchart LR
A[1. load_custom_query] --> B[2. retrain_model] --> C[3. update_production_model] --> D[4. export_data_to_postgres]
A -.-> Database[(Database)]
B -.-> MLFlow[MLFlow]
C -.-> MLFlow[MLFlow]
D -.-> PostgreSQL[(PostgreSQL)]
```
## 📋 Prerequisites
- Python 3.11+