SIENTIAPDE-1084

Enhance README.md by adding detailed system architecture and data processing flow diagrams using Mermaid syntax, improving clarity on the Scouter system's components and interactions.

README.md

@@ -23,32 +23,144 @@ A high-performance, scalable data processing system built on Temporal.io for ind
 
 The Scouter system uses a Temporal-based workflow architecture with clear separation of concerns:
 
+## System Architecture Diagram
+
+```mermaid
+graph TB
+    %% External Data Sources
+    Kafka[Kafka Topics]
+    OPC[OPC UA Servers]
+    
+    %% Temporal Infrastructure
+    subgraph "Temporal Cluster"
+        TC[Temporal Client]
+        TW[Temporal Workers]
+        TQ1[scouter-queue]
+        TQ2[fake_data-queue]
+    end
+    
+    %% Scouter Application
+    subgraph "Scouter Application"
+        Worker[Worker Process]
+        
+        %% Workflows
+        subgraph "Workflows"
+            ScouterWF[Scouter Workflow<br/>Data Orchestration]
+            CoreWF[CoreScouter Workflow<br/>Data Processing]
+            FakeWF[FakeData Workflow<br/>Test Data Generation]
+        end
+        
+        %% Activities Layer
+        subgraph "Activities Layer"
+            ActMain[Activities<br/>Unified Interface]
+            RedisAct[Redis Activities<br/>Caching & Timestamps]
+            MongoAct[MongoDB Activities<br/>Data Retrieval]
+            PostgresAct[PostgreSQL Activities<br/>Data Persistence]
+            GatesAct[Quality Gates<br/>Data Validation]
+            FakerAct[Faker Activities<br/>Test Data Generation]
+        end
+    end
+    
+    %% Data Storage Layer
+    subgraph "Data Storage"
+        MongoDB[(MongoDB<br/>Raw Data Collections)]
+        Redis[(Redis<br/>Cache & Timestamps)]
+        PostgreSQL[(PostgreSQL<br/>Processed Data)]
+    end
+    
+    %% Monitoring
+    subgraph "Monitoring & Observability"
+        Prometheus[Prometheus Metrics]
+        Logs[Application Logs]
+        Notifications[Notification System]
+    end
+    
+    %% Data Flow Connections
+    Kafka -->|Raw Data| MongoDB
+    OPC -->|Sensor Data| MongoDB
+    
+    %% Worker to Temporal
+    Worker --> TC
+    TC --> TQ1
+    TC --> TQ2
+    TQ1 --> ScouterWF
+    TQ2 --> FakeWF
+    
+    %% Workflow Orchestration
+    ScouterWF -->|Child Workflow| CoreWF
+    ScouterWF --> RedisAct
+    ScouterWF --> MongoAct
+    
+    %% Core Processing Flow
+    CoreWF --> GatesAct
+    CoreWF --> RedisAct
+    CoreWF --> PostgresAct
+    
+    %% Test Data Flow
+    FakeWF --> FakerAct
+    FakerAct -->|Generate| Kafka
+    
+    %% Activities to Data Sources
+    ActMain --> RedisAct
+    ActMain --> MongoAct
+    ActMain --> PostgresAct
+    ActMain --> GatesAct
+    
+    RedisAct <--> Redis
+    MongoAct --> MongoDB
+    PostgresAct --> PostgreSQL
+    
+    %% Monitoring Connections
+    Worker --> Prometheus
+    ActMain --> Logs
+    ActMain --> Notifications
+    
+    %% Styling
+    classDef workflow fill:#e1f5fe,stroke:#01579b,stroke-width:2px
+    classDef activity fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
+    classDef storage fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
+    classDef external fill:#fff3e0,stroke:#e65100,stroke-width:2px
+    classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px
+    
+    class ScouterWF,CoreWF,FakeWF workflow
+    class ActMain,RedisAct,MongoAct,PostgresAct,GatesAct,FakerAct activity
+    class MongoDB,Redis,PostgreSQL storage
+    class Kafka,OPC external
+    class Prometheus,Logs,Notifications monitoring
 ```
-┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
+
-│   Main Worker   │    │  Temporal Client │    │   Task Queues   │
+## Data Processing Flow
-│                 │◄──►│                  │◄──►│                 │
+
-│ - Metrics Server│    │ - Namespace Mgmt │    │ - scouter-queue │
+```mermaid
-│ - Notifications │    │ - Runtime Config │    │ - fake_data-queue│
+sequenceDiagram
-│ - Lifecycle     │    │ - Connection     │    │ - Auto-scaling  │
+    participant K as Kafka/OPC
-└─────────────────┘    └──────────────────┘    └─────────────────┘
+    participant M as MongoDB
-                                │
+    participant SW as Scouter Workflow
-                                ▼
+    participant R as Redis
-                       ┌──────────────────┐    ┌─────────────────┐
+    participant CW as CoreScouter Workflow
-                       │  Workflow Layer  │    │  Activity Layer │
+    participant G as Quality Gates
-                       │                  │    │                 │
+    participant P as PostgreSQL
-                       │ - Scouter        │    │ - Data Quality  │
+    
-                       │ - CoreScouter    │    │ - Aggregation   │
+    %% Data Ingestion
-                       │ - FakeData       │    │ - Storage Ops   │
+    K->>M: Raw sensor data
-                       └──────────────────┘    └─────────────────┘
+    
-                                │
+    %% Workflow Execution
-                                ▼
+    SW->>R: Get last timestamp
-                       ┌──────────────────┐    ┌─────────────────┐
+    SW->>M: Load new data since timestamp
-                       │  Data Services   │    │  External       │
+    SW->>R: Update last timestamp
-                       │                  │    │  Systems        │
+    SW->>CW: Process data (child workflow)
-                       │ - PostgreSQL     │    │ - Kafka Topics  │
+    
-                       │ - Redis Cache    │    │ - OPC Servers   │
+    %% Core Processing
-                       │ - MongoDB        │    │ - Prometheus    │
+    CW->>G: Apply quality filters
-                       └──────────────────┘    └─────────────────┘
+    G->>G: Validate data ranges
+    G->>G: Remove null values
+    CW->>G: Aggregate data by tags
+    CW->>R: Store grouped data (TTL)
+    CW->>P: Export to PostgreSQL
+    
+    %% Monitoring
+    CW->>CW: Record metrics
+    P-->>SW: Processing complete
 ```
 
 ### Key Components
@@ -100,6 +212,8 @@ The **Scouter** workflow is the main entry point for data processing pipelines.
 }
 ```
 
+
+
 ### 2. CoreScouter Workflow (`core_scouter.py`)
 
 The **CoreScouter** workflow implements the core data processing pipeline for industrial time-series data. It handles data quality validation, aggregation, and export operations.