From cbad5df8d2ab7bd3ee5096dec1911dd722f324bc Mon Sep 17 00:00:00 2001
From: vitor-aignosi <vitor.santos@aignosi.com.br>
Date: Fri, 29 Aug 2025 16:27:21 -0300
Subject: [PATCH] SIENTIAPDE-1084  Update README.md to include a new section on
 Scouter Workflow Architecture, detailing the incremental data processing
 steps and key components, while removing outdated system architecture
 diagrams for improved clarity and relevance.

---
 README.md | 161 +++++++-----------------------------------------------
 1 file changed, 21 insertions(+), 140 deletions(-)

diff --git a/README.md b/README.md
index 32de4f2..84437a2 100644
--- a/README.md
+++ b/README.md
@@ -23,146 +23,6 @@ A high-performance, scalable data processing system built on Temporal.io for ind
 
 The Scouter system uses a Temporal-based workflow architecture with clear separation of concerns:
 
-## System Architecture Diagram
-
-```mermaid
-graph TB
-    %% External Data Sources
-    Kafka[Kafka Topics]
-    OPC[OPC UA Servers]
-    
-    %% Temporal Infrastructure
-    subgraph "Temporal Cluster"
-        TC[Temporal Client]
-        TW[Temporal Workers]
-        TQ1[scouter-queue]
-        TQ2[fake_data-queue]
-    end
-    
-    %% Scouter Application
-    subgraph "Scouter Application"
-        Worker[Worker Process]
-        
-        %% Workflows
-        subgraph "Workflows"
-            ScouterWF[Scouter Workflow<br/>Data Orchestration]
-            CoreWF[CoreScouter Workflow<br/>Data Processing]
-            FakeWF[FakeData Workflow<br/>Test Data Generation]
-        end
-        
-        %% Activities Layer
-        subgraph "Activities Layer"
-            ActMain[Activities<br/>Unified Interface]
-            RedisAct[Redis Activities<br/>Caching & Timestamps]
-            MongoAct[MongoDB Activities<br/>Data Retrieval]
-            PostgresAct[PostgreSQL Activities<br/>Data Persistence]
-            GatesAct[Quality Gates<br/>Data Validation]
-            FakerAct[Faker Activities<br/>Test Data Generation]
-        end
-    end
-    
-    %% Data Storage Layer
-    subgraph "Data Storage"
-        MongoDB[(MongoDB<br/>Raw Data Collections)]
-        Redis[(Redis<br/>Cache & Timestamps)]
-        PostgreSQL[(PostgreSQL<br/>Processed Data)]
-    end
-    
-    %% Monitoring
-    subgraph "Monitoring & Observability"
-        Prometheus[Prometheus Metrics]
-        Logs[Application Logs]
-        Notifications[Notification System]
-    end
-    
-    %% Data Flow Connections
-    Kafka -->|Raw Data| MongoDB
-    OPC -->|Sensor Data| MongoDB
-    
-    %% Worker to Temporal
-    Worker --> TC
-    TC --> TQ1
-    TC --> TQ2
-    TQ1 --> ScouterWF
-    TQ2 --> FakeWF
-    
-    %% Workflow Orchestration
-    ScouterWF -->|Child Workflow| CoreWF
-    ScouterWF --> RedisAct
-    ScouterWF --> MongoAct
-    
-    %% Core Processing Flow
-    CoreWF --> GatesAct
-    CoreWF --> RedisAct
-    CoreWF --> PostgresAct
-    
-    %% Test Data Flow
-    FakeWF --> FakerAct
-    FakerAct -->|Generate| Kafka
-    
-    %% Activities to Data Sources
-    ActMain --> RedisAct
-    ActMain --> MongoAct
-    ActMain --> PostgresAct
-    ActMain --> GatesAct
-    
-    RedisAct <--> Redis
-    MongoAct --> MongoDB
-    PostgresAct --> PostgreSQL
-    
-    %% Monitoring Connections
-    Worker --> Prometheus
-    ActMain --> Logs
-    ActMain --> Notifications
-    
-    %% Styling
-    classDef workflow fill:#e1f5fe,stroke:#01579b,stroke-width:2px
-    classDef activity fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
-    classDef storage fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
-    classDef external fill:#fff3e0,stroke:#e65100,stroke-width:2px
-    classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px
-    
-    class ScouterWF,CoreWF,FakeWF workflow
-    class ActMain,RedisAct,MongoAct,PostgresAct,GatesAct,FakerAct activity
-    class MongoDB,Redis,PostgreSQL storage
-    class Kafka,OPC external
-    class Prometheus,Logs,Notifications monitoring
-```
-
-## Data Processing Flow
-
-```mermaid
-sequenceDiagram
-    participant K as Kafka/OPC
-    participant M as MongoDB
-    participant SW as Scouter Workflow
-    participant R as Redis
-    participant CW as CoreScouter Workflow
-    participant G as Quality Gates
-    participant P as PostgreSQL
-    
-    %% Data Ingestion
-    K->>M: Raw sensor data
-    
-    %% Workflow Execution
-    SW->>R: Get last timestamp
-    SW->>M: Load new data since timestamp
-    SW->>R: Update last timestamp
-    SW->>CW: Process data (child workflow)
-    
-    %% Core Processing
-    CW->>G: Apply quality filters
-    G->>G: Validate data ranges
-    G->>G: Remove null values
-    CW->>G: Aggregate data by tags
-    CW->>R: Store grouped data (TTL)
-    CW->>P: Export to PostgreSQL
-    
-    %% Monitoring
-    CW->>CW: Record metrics
-    P-->>SW: Processing complete
-```
-
 ### Key Components
 
 - **Worker**: Main application orchestrator managing Temporal workers and task queues
@@ -212,6 +72,27 @@ The **Scouter** workflow is the main entry point for data processing pipelines.
 }
 ```
 
+## Scouter Workflow Architecture
+
+## Fluxo do Scouter Workflow
+
+```mermaid
+flowchart TD
+    A[1. get_last_data_timestamp] --> B[2. load_latest_data]
+    B --> C[3. put_last_data_timestamp] 
+    C --> D[4. core_scouter ⬛]
+    
+    A -.-> Redis[(Redis)]
+    B -.-> MongoDB[(MongoDB)]
+    C -.-> Redis
+```
+
+### Key Components
+
+- **Scouter Workflow**: Main orchestrator for incremental data processing
+- **Redis Activities**: Timestamp management for incremental processing
+- **MongoDB Activities**: Raw data retrieval from collections
+- **CoreScouter (Black Box)**: Complete data processing pipeline including quality gates, aggregation, and PostgreSQL export
 
 
 ### 2. CoreScouter Workflow (`core_scouter.py`)