gitea_admin/sientia-dataops-scouter_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1084

Browse Source 
Enhance README.md by adding detailed system architecture and data processing flow diagrams using Mermaid syntax, improving clarity on the Scouter system's components and interactions.
This commit is contained in:
vitor-aignosi
2025-08-29 16:20:07 -03:00
parent beb06d257f
commit fe6133f31a
1 changed files with 139 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

164
README.md
View File

@@ -23,32 +23,144 @@ A high-performance, scalable data processing system built on Temporal.io for ind
The Scouter system uses a Temporal-based workflow architecture with clear separation of concerns:
## System Architecture Diagram
```mermaid
graph TB
%% External Data Sources
Kafka[Kafka Topics]
OPC[OPC UA Servers]
%% Temporal Infrastructure
subgraph "Temporal Cluster"
TC[Temporal Client]
TW[Temporal Workers]
TQ1[scouter-queue]
TQ2[fake_data-queue]
end
%% Scouter Application
subgraph "Scouter Application"
Worker[Worker Process]
%% Workflows
subgraph "Workflows"
ScouterWF[Scouter Workflow<br/>Data Orchestration]
CoreWF[CoreScouter Workflow<br/>Data Processing]
FakeWF[FakeData Workflow<br/>Test Data Generation]
end
%% Activities Layer
subgraph "Activities Layer"
ActMain[Activities<br/>Unified Interface]
RedisAct[Redis Activities<br/>Caching & Timestamps]
MongoAct[MongoDB Activities<br/>Data Retrieval]
PostgresAct[PostgreSQL Activities<br/>Data Persistence]
GatesAct[Quality Gates<br/>Data Validation]
FakerAct[Faker Activities<br/>Test Data Generation]
end
end
%% Data Storage Layer
subgraph "Data Storage"
MongoDB[(MongoDB<br/>Raw Data Collections)]
Redis[(Redis<br/>Cache & Timestamps)]
PostgreSQL[(PostgreSQL<br/>Processed Data)]
end
%% Monitoring
subgraph "Monitoring & Observability"
Prometheus[Prometheus Metrics]
Logs[Application Logs]
Notifications[Notification System]
end
%% Data Flow Connections
Kafka -->|Raw Data| MongoDB
OPC -->|Sensor Data| MongoDB
%% Worker to Temporal
Worker --> TC
TC --> TQ1
TC --> TQ2
TQ1 --> ScouterWF
TQ2 --> FakeWF
%% Workflow Orchestration
ScouterWF -->|Child Workflow| CoreWF
ScouterWF --> RedisAct
ScouterWF --> MongoAct
%% Core Processing Flow
CoreWF --> GatesAct
CoreWF --> RedisAct
CoreWF --> PostgresAct
%% Test Data Flow
FakeWF --> FakerAct
FakerAct -->|Generate| Kafka
%% Activities to Data Sources
ActMain --> RedisAct
ActMain --> MongoAct
ActMain --> PostgresAct
ActMain --> GatesAct
RedisAct <--> Redis
MongoAct --> MongoDB
PostgresAct --> PostgreSQL
%% Monitoring Connections
Worker --> Prometheus
ActMain --> Logs
ActMain --> Notifications
%% Styling
classDef workflow fill:#e1f5fe,stroke:#01579b,stroke-width:2px
classDef activity fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
classDef storage fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
classDef external fill:#fff3e0,stroke:#e65100,stroke-width:2px
classDef monitoring fill:#fce4ec,stroke:#880e4f,stroke-width:2px
class ScouterWF,CoreWF,FakeWF workflow
class ActMain,RedisAct,MongoAct,PostgresAct,GatesAct,FakerAct activity
class MongoDB,Redis,PostgreSQL storage
class Kafka,OPC external
class Prometheus,Logs,Notifications monitoring
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Main Worker │ │ Temporal Client │ │ Task Queues │
│ │◄──►│ │◄──►│ │
│ - Metrics Server│ │ - Namespace Mgmt │ │ - scouter-queue │
│ - Notifications │ │ - Runtime Config │ │ - fake_data-queue│
│ - Lifecycle │ │ - Connection │ │ - Auto-scaling │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│
▼
┌──────────────────┐ ┌─────────────────┐
│ Workflow Layer │ │ Activity Layer │
│ │ │ │
│ - Scouter │ │ - Data Quality │
│ - CoreScouter │ │ - Aggregation │
│ - FakeData │ │ - Storage Ops │
└──────────────────┘ └─────────────────┘
│
▼
┌──────────────────┐ ┌─────────────────┐
│ Data Services │ │ External │
│ │ │ Systems │
│ - PostgreSQL │ │ - Kafka Topics │
│ - Redis Cache │ │ - OPC Servers │
│ - MongoDB │ │ - Prometheus │
└──────────────────┘ └─────────────────┘
## Data Processing Flow
```mermaid
sequenceDiagram
participant K as Kafka/OPC
participant M as MongoDB
participant SW as Scouter Workflow
participant R as Redis
participant CW as CoreScouter Workflow
participant G as Quality Gates
participant P as PostgreSQL
%% Data Ingestion
K->>M: Raw sensor data
%% Workflow Execution
SW->>R: Get last timestamp
SW->>M: Load new data since timestamp
SW->>R: Update last timestamp
SW->>CW: Process data (child workflow)
%% Core Processing
CW->>G: Apply quality filters
G->>G: Validate data ranges
G->>G: Remove null values
CW->>G: Aggregate data by tags
CW->>R: Store grouped data (TTL)
CW->>P: Export to PostgreSQL
%% Monitoring
CW->>CW: Record metrics
P-->>SW: Processing complete
```
### Key Components
@@ -100,6 +212,8 @@ The **Scouter** workflow is the main entry point for data processing pipelines.
}
```
### 2. CoreScouter Workflow (`core_scouter.py`)
The **CoreScouter** workflow implements the core data processing pipeline for industrial time-series data. It handles data quality validation, aggregation, and export operations.