From fe6133f31a41fa0469e92ac487d0646650540af5 Mon Sep 17 00:00:00 2001 From: vitor-aignosi Date: Fri, 29 Aug 2025 16:20:07 -0300 Subject: [PATCH] SIENTIAPDE-1084 Enhance README.md by adding detailed system architecture and data processing flow diagrams using Mermaid syntax, improving clarity on the Scouter system's components and interactions. --- README.md | 164 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 139 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 72aea2f..32de4f2 100644 --- a/README.md +++ b/README.md @@ -23,32 +23,144 @@ A high-performance, scalable data processing system built on Temporal.io for ind The Scouter system uses a Temporal-based workflow architecture with clear separation of concerns: +## System Architecture Diagram + +```mermaid +graph TB + %% External Data Sources + Kafka[Kafka Topics] + OPC[OPC UA Servers] + + %% Temporal Infrastructure + subgraph "Temporal Cluster" + TC[Temporal Client] + TW[Temporal Workers] + TQ1[scouter-queue] + TQ2[fake_data-queue] + end + + %% Scouter Application + subgraph "Scouter Application" + Worker[Worker Process] + + %% Workflows + subgraph "Workflows" + ScouterWF[Scouter Workflow
Data Orchestration] + CoreWF[CoreScouter Workflow
Data Processing] + FakeWF[FakeData Workflow
Test Data Generation] + end + + %% Activities Layer + subgraph "Activities Layer" + ActMain[Activities
Unified Interface] + RedisAct[Redis Activities
Caching & Timestamps] + MongoAct[MongoDB Activities
Data Retrieval] + PostgresAct[PostgreSQL Activities
Data Persistence] + GatesAct[Quality Gates
Data Validation] + FakerAct[Faker Activities
Test Data Generation] + end + end + + %% Data Storage Layer + subgraph "Data Storage" + MongoDB[(MongoDB
Raw Data Collections)] + Redis[(Redis
Cache & Timestamps)] + PostgreSQL[(PostgreSQL
Processed Data)] + end + + %% Monitoring + subgraph "Monitoring & Observability" + Prometheus[Prometheus Metrics] + Logs[Application Logs] + Notifications[Notification System] + end + + %% Data Flow Connections + Kafka -->|Raw Data| MongoDB + OPC -->|Sensor Data| MongoDB + + %% Worker to Temporal + Worker --> TC + TC --> TQ1 + TC --> TQ2 + TQ1 --> ScouterWF + TQ2 --> FakeWF + + %% Workflow Orchestration + ScouterWF -->|Child Workflow| CoreWF + ScouterWF --> RedisAct + ScouterWF --> MongoAct + + %% Core Processing Flow + CoreWF --> GatesAct + CoreWF --> RedisAct + CoreWF --> PostgresAct + + %% Test Data Flow + FakeWF --> FakerAct + FakerAct -->|Generate| Kafka + + %% Activities to Data Sources + ActMain --> RedisAct + ActMain --> MongoAct + ActMain --> PostgresAct + ActMain --> GatesAct + + RedisAct <--> Redis + MongoAct --> MongoDB + PostgresAct --> PostgreSQL + + %% Monitoring Connections + Worker --> Prometheus + ActMain --> Logs + ActMain --> Notifications + + %% Styling + classDef workflow fill:#e1f5fe,stroke:#01579b,stroke-width:2px + classDef activity fill:#f3e5f5,stroke:#4a148c,stroke-width:2px + classDef storage fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px + classDef external fill:#fff3e0,stroke:#e65100,stroke-width:2px + classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px + + class ScouterWF,CoreWF,FakeWF workflow + class ActMain,RedisAct,MongoAct,PostgresAct,GatesAct,FakerAct activity + class MongoDB,Redis,PostgreSQL storage + class Kafka,OPC external + class Prometheus,Logs,Notifications monitoring ``` -┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ -│ Main Worker │ │ Temporal Client │ │ Task Queues │ -│ │◄──►│ │◄──►│ │ -│ - Metrics Server│ │ - Namespace Mgmt │ │ - scouter-queue │ -│ - Notifications │ │ - Runtime Config │ │ - fake_data-queue│ -│ - Lifecycle │ │ - Connection │ │ - Auto-scaling │ -└─────────────────┘ └──────────────────┘ └─────────────────┘ - │ - ▼ - ┌──────────────────┐ ┌─────────────────┐ - │ Workflow Layer │ │ Activity Layer │ - │ │ │ │ - │ - Scouter │ │ - Data Quality │ - │ - CoreScouter │ │ - Aggregation │ - │ - FakeData │ │ - Storage Ops │ - └──────────────────┘ └─────────────────┘ - │ - ▼ - ┌──────────────────┐ ┌─────────────────┐ - │ Data Services │ │ External │ - │ │ │ Systems │ - │ - PostgreSQL │ │ - Kafka Topics │ - │ - Redis Cache │ │ - OPC Servers │ - │ - MongoDB │ │ - Prometheus │ - └──────────────────┘ └─────────────────┘ + +## Data Processing Flow + +```mermaid +sequenceDiagram + participant K as Kafka/OPC + participant M as MongoDB + participant SW as Scouter Workflow + participant R as Redis + participant CW as CoreScouter Workflow + participant G as Quality Gates + participant P as PostgreSQL + + %% Data Ingestion + K->>M: Raw sensor data + + %% Workflow Execution + SW->>R: Get last timestamp + SW->>M: Load new data since timestamp + SW->>R: Update last timestamp + SW->>CW: Process data (child workflow) + + %% Core Processing + CW->>G: Apply quality filters + G->>G: Validate data ranges + G->>G: Remove null values + CW->>G: Aggregate data by tags + CW->>R: Store grouped data (TTL) + CW->>P: Export to PostgreSQL + + %% Monitoring + CW->>CW: Record metrics + P-->>SW: Processing complete ``` ### Key Components @@ -100,6 +212,8 @@ The **Scouter** workflow is the main entry point for data processing pipelines. } ``` + + ### 2. CoreScouter Workflow (`core_scouter.py`) The **CoreScouter** workflow implements the core data processing pipeline for industrial time-series data. It handles data quality validation, aggregation, and export operations.