diff --git a/README.md b/README.md
index 72aea2f..32de4f2 100644
--- a/README.md
+++ b/README.md
@@ -23,32 +23,144 @@ A high-performance, scalable data processing system built on Temporal.io for ind
The Scouter system uses a Temporal-based workflow architecture with clear separation of concerns:
+## System Architecture Diagram
+
+```mermaid
+graph TB
+ %% External Data Sources
+ Kafka[Kafka Topics]
+ OPC[OPC UA Servers]
+
+ %% Temporal Infrastructure
+ subgraph "Temporal Cluster"
+ TC[Temporal Client]
+ TW[Temporal Workers]
+ TQ1[scouter-queue]
+ TQ2[fake_data-queue]
+ end
+
+ %% Scouter Application
+ subgraph "Scouter Application"
+ Worker[Worker Process]
+
+ %% Workflows
+ subgraph "Workflows"
+ ScouterWF[Scouter Workflow
Data Orchestration]
+ CoreWF[CoreScouter Workflow
Data Processing]
+ FakeWF[FakeData Workflow
Test Data Generation]
+ end
+
+ %% Activities Layer
+ subgraph "Activities Layer"
+ ActMain[Activities
Unified Interface]
+ RedisAct[Redis Activities
Caching & Timestamps]
+ MongoAct[MongoDB Activities
Data Retrieval]
+ PostgresAct[PostgreSQL Activities
Data Persistence]
+ GatesAct[Quality Gates
Data Validation]
+ FakerAct[Faker Activities
Test Data Generation]
+ end
+ end
+
+ %% Data Storage Layer
+ subgraph "Data Storage"
+ MongoDB[(MongoDB
Raw Data Collections)]
+ Redis[(Redis
Cache & Timestamps)]
+ PostgreSQL[(PostgreSQL
Processed Data)]
+ end
+
+ %% Monitoring
+ subgraph "Monitoring & Observability"
+ Prometheus[Prometheus Metrics]
+ Logs[Application Logs]
+ Notifications[Notification System]
+ end
+
+ %% Data Flow Connections
+ Kafka -->|Raw Data| MongoDB
+ OPC -->|Sensor Data| MongoDB
+
+ %% Worker to Temporal
+ Worker --> TC
+ TC --> TQ1
+ TC --> TQ2
+ TQ1 --> ScouterWF
+ TQ2 --> FakeWF
+
+ %% Workflow Orchestration
+ ScouterWF -->|Child Workflow| CoreWF
+ ScouterWF --> RedisAct
+ ScouterWF --> MongoAct
+
+ %% Core Processing Flow
+ CoreWF --> GatesAct
+ CoreWF --> RedisAct
+ CoreWF --> PostgresAct
+
+ %% Test Data Flow
+ FakeWF --> FakerAct
+ FakerAct -->|Generate| Kafka
+
+ %% Activities to Data Sources
+ ActMain --> RedisAct
+ ActMain --> MongoAct
+ ActMain --> PostgresAct
+ ActMain --> GatesAct
+
+ RedisAct <--> Redis
+ MongoAct --> MongoDB
+ PostgresAct --> PostgreSQL
+
+ %% Monitoring Connections
+ Worker --> Prometheus
+ ActMain --> Logs
+ ActMain --> Notifications
+
+ %% Styling
+ classDef workflow fill:#e1f5fe,stroke:#01579b,stroke-width:2px
+ classDef activity fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
+ classDef storage fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
+ classDef external fill:#fff3e0,stroke:#e65100,stroke-width:2px
+ classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px
+
+ class ScouterWF,CoreWF,FakeWF workflow
+ class ActMain,RedisAct,MongoAct,PostgresAct,GatesAct,FakerAct activity
+ class MongoDB,Redis,PostgreSQL storage
+ class Kafka,OPC external
+ class Prometheus,Logs,Notifications monitoring
```
-┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
-│ Main Worker │ │ Temporal Client │ │ Task Queues │
-│ │◄──►│ │◄──►│ │
-│ - Metrics Server│ │ - Namespace Mgmt │ │ - scouter-queue │
-│ - Notifications │ │ - Runtime Config │ │ - fake_data-queue│
-│ - Lifecycle │ │ - Connection │ │ - Auto-scaling │
-└─────────────────┘ └──────────────────┘ └─────────────────┘
- │
- ▼
- ┌──────────────────┐ ┌─────────────────┐
- │ Workflow Layer │ │ Activity Layer │
- │ │ │ │
- │ - Scouter │ │ - Data Quality │
- │ - CoreScouter │ │ - Aggregation │
- │ - FakeData │ │ - Storage Ops │
- └──────────────────┘ └─────────────────┘
- │
- ▼
- ┌──────────────────┐ ┌─────────────────┐
- │ Data Services │ │ External │
- │ │ │ Systems │
- │ - PostgreSQL │ │ - Kafka Topics │
- │ - Redis Cache │ │ - OPC Servers │
- │ - MongoDB │ │ - Prometheus │
- └──────────────────┘ └─────────────────┘
+
+## Data Processing Flow
+
+```mermaid
+sequenceDiagram
+ participant K as Kafka/OPC
+ participant M as MongoDB
+ participant SW as Scouter Workflow
+ participant R as Redis
+ participant CW as CoreScouter Workflow
+ participant G as Quality Gates
+ participant P as PostgreSQL
+
+ %% Data Ingestion
+ K->>M: Raw sensor data
+
+ %% Workflow Execution
+ SW->>R: Get last timestamp
+ SW->>M: Load new data since timestamp
+ SW->>R: Update last timestamp
+ SW->>CW: Process data (child workflow)
+
+ %% Core Processing
+ CW->>G: Apply quality filters
+ G->>G: Validate data ranges
+ G->>G: Remove null values
+ CW->>G: Aggregate data by tags
+ CW->>R: Store grouped data (TTL)
+ CW->>P: Export to PostgreSQL
+
+ %% Monitoring
+ CW->>CW: Record metrics
+ P-->>SW: Processing complete
```
### Key Components
@@ -100,6 +212,8 @@ The **Scouter** workflow is the main entry point for data processing pipelines.
}
```
+
+
### 2. CoreScouter Workflow (`core_scouter.py`)
The **CoreScouter** workflow implements the core data processing pipeline for industrial time-series data. It handles data quality validation, aggregation, and export operations.