SIENTIAPDE-1084
Update README.md to include a new section on Scouter Workflow Architecture, detailing the incremental data processing steps and key components, while removing outdated system architecture diagrams for improved clarity and relevance.
This commit is contained in:
vitor-aignosi
161
README.md
161
README.md
@@ -23,146 +23,6 @@ A high-performance, scalable data processing system built on Temporal.io for ind
|
||||
|
||||
The Scouter system uses a Temporal-based workflow architecture with clear separation of concerns:
|
||||
|
||||
## System Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
%% External Data Sources
|
||||
Kafka[Kafka Topics]
|
||||
OPC[OPC UA Servers]
|
||||
|
||||
%% Temporal Infrastructure
|
||||
subgraph "Temporal Cluster"
|
||||
TC[Temporal Client]
|
||||
TW[Temporal Workers]
|
||||
TQ1[scouter-queue]
|
||||
TQ2[fake_data-queue]
|
||||
end
|
||||
|
||||
%% Scouter Application
|
||||
subgraph "Scouter Application"
|
||||
Worker[Worker Process]
|
||||
|
||||
%% Workflows
|
||||
subgraph "Workflows"
|
||||
ScouterWF[Scouter Workflow<br/>Data Orchestration]
|
||||
CoreWF[CoreScouter Workflow<br/>Data Processing]
|
||||
FakeWF[FakeData Workflow<br/>Test Data Generation]
|
||||
end
|
||||
|
||||
%% Activities Layer
|
||||
subgraph "Activities Layer"
|
||||
ActMain[Activities<br/>Unified Interface]
|
||||
RedisAct[Redis Activities<br/>Caching & Timestamps]
|
||||
MongoAct[MongoDB Activities<br/>Data Retrieval]
|
||||
PostgresAct[PostgreSQL Activities<br/>Data Persistence]
|
||||
GatesAct[Quality Gates<br/>Data Validation]
|
||||
FakerAct[Faker Activities<br/>Test Data Generation]
|
||||
end
|
||||
end
|
||||
|
||||
%% Data Storage Layer
|
||||
subgraph "Data Storage"
|
||||
MongoDB[(MongoDB<br/>Raw Data Collections)]
|
||||
Redis[(Redis<br/>Cache & Timestamps)]
|
||||
PostgreSQL[(PostgreSQL<br/>Processed Data)]
|
||||
end
|
||||
|
||||
%% Monitoring
|
||||
subgraph "Monitoring & Observability"
|
||||
Prometheus[Prometheus Metrics]
|
||||
Logs[Application Logs]
|
||||
Notifications[Notification System]
|
||||
end
|
||||
|
||||
%% Data Flow Connections
|
||||
Kafka -->|Raw Data| MongoDB
|
||||
OPC -->|Sensor Data| MongoDB
|
||||
|
||||
%% Worker to Temporal
|
||||
Worker --> TC
|
||||
TC --> TQ1
|
||||
TC --> TQ2
|
||||
TQ1 --> ScouterWF
|
||||
TQ2 --> FakeWF
|
||||
|
||||
%% Workflow Orchestration
|
||||
ScouterWF -->|Child Workflow| CoreWF
|
||||
ScouterWF --> RedisAct
|
||||
ScouterWF --> MongoAct
|
||||
|
||||
%% Core Processing Flow
|
||||
CoreWF --> GatesAct
|
||||
CoreWF --> RedisAct
|
||||
CoreWF --> PostgresAct
|
||||
|
||||
%% Test Data Flow
|
||||
FakeWF --> FakerAct
|
||||
FakerAct -->|Generate| Kafka
|
||||
|
||||
%% Activities to Data Sources
|
||||
ActMain --> RedisAct
|
||||
ActMain --> MongoAct
|
||||
ActMain --> PostgresAct
|
||||
ActMain --> GatesAct
|
||||
|
||||
RedisAct <--> Redis
|
||||
MongoAct --> MongoDB
|
||||
PostgresAct --> PostgreSQL
|
||||
|
||||
%% Monitoring Connections
|
||||
Worker --> Prometheus
|
||||
ActMain --> Logs
|
||||
ActMain --> Notifications
|
||||
|
||||
%% Styling
|
||||
classDef workflow fill:#e1f5fe,stroke:#01579b,stroke-width:2px
|
||||
classDef activity fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
|
||||
classDef storage fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
|
||||
classDef external fill:#fff3e0,stroke:#e65100,stroke-width:2px
|
||||
classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px
|
||||
|
||||
class ScouterWF,CoreWF,FakeWF workflow
|
||||
class ActMain,RedisAct,MongoAct,PostgresAct,GatesAct,FakerAct activity
|
||||
class MongoDB,Redis,PostgreSQL storage
|
||||
class Kafka,OPC external
|
||||
class Prometheus,Logs,Notifications monitoring
|
||||
```
|
||||
|
||||
## Data Processing Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant K as Kafka/OPC
|
||||
participant M as MongoDB
|
||||
participant SW as Scouter Workflow
|
||||
participant R as Redis
|
||||
participant CW as CoreScouter Workflow
|
||||
participant G as Quality Gates
|
||||
participant P as PostgreSQL
|
||||
|
||||
%% Data Ingestion
|
||||
K->>M: Raw sensor data
|
||||
|
||||
%% Workflow Execution
|
||||
SW->>R: Get last timestamp
|
||||
SW->>M: Load new data since timestamp
|
||||
SW->>R: Update last timestamp
|
||||
SW->>CW: Process data (child workflow)
|
||||
|
||||
%% Core Processing
|
||||
CW->>G: Apply quality filters
|
||||
G->>G: Validate data ranges
|
||||
G->>G: Remove null values
|
||||
CW->>G: Aggregate data by tags
|
||||
CW->>R: Store grouped data (TTL)
|
||||
CW->>P: Export to PostgreSQL
|
||||
|
||||
%% Monitoring
|
||||
CW->>CW: Record metrics
|
||||
P-->>SW: Processing complete
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
- **Worker**: Main application orchestrator managing Temporal workers and task queues
|
||||
@@ -212,6 +72,27 @@ The **Scouter** workflow is the main entry point for data processing pipelines.
|
||||
}
|
||||
```
|
||||
|
||||
## Scouter Workflow Architecture
|
||||
|
||||
## Fluxo do Scouter Workflow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[1. get_last_data_timestamp] --> B[2. load_latest_data]
|
||||
B --> C[3. put_last_data_timestamp]
|
||||
C --> D[4. core_scouter ⬛]
|
||||
|
||||
A -.-> Redis[(Redis)]
|
||||
B -.-> MongoDB[(MongoDB)]
|
||||
C -.-> Redis
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
- **Scouter Workflow**: Main orchestrator for incremental data processing
|
||||
- **Redis Activities**: Timestamp management for incremental processing
|
||||
- **MongoDB Activities**: Raw data retrieval from collections
|
||||
- **CoreScouter (Black Box)**: Complete data processing pipeline including quality gates, aggregation, and PostgreSQL export
|
||||
|
||||
|
||||
### 2. CoreScouter Workflow (`core_scouter.py`)
|
||||
|
||||
Reference in New Issue
Block a user