SIENTIAPDE-1184
docs: update README.md to provide comprehensive overview of SIENTIA DataOps Orchestrator Temporal - Added detailed project overview, goals, and architecture sections. - Documented main workflows including orchestrator, alerts, and reports workflows with input parameters and functionalities. - Included environment variable configurations for Redis, MongoDB, PostgreSQL, Couchbase, email, Temporal, application, and Kafka. - Described deployment strategies and monitoring capabilities, including Prometheus metrics. - Enhanced usage instructions for triggering workflows and logging operations.
This commit is contained in:
vitor-aignosi
186
README.md
186
README.md
@@ -1,4 +1,188 @@
|
||||
#PR shortcut
|
||||
# SIENTIA DataOps Orchestrator Temporal
|
||||
|
||||
## Overview
|
||||
|
||||
The SIENTIA DataOps Orchestrator Temporal is a comprehensive workflow orchestration system built on Temporal.io that manages data pipelines, notifications, and system orchestration for the SIENTIA platform. It provides automated scheduling, monitoring, and execution of data processing workflows with integrated alerting and reporting capabilities.
|
||||
|
||||
## Project Goals
|
||||
|
||||
- **Pipeline Orchestration**: Automate the deployment and management of data processing pipelines
|
||||
- **Notification Management**: Handle real-time alerts and scheduled reports for system events
|
||||
- **Resource Management**: Manage OPC server slots and data ingestion resources
|
||||
- **Workflow Automation**: Coordinate complex workflows across multiple services and databases
|
||||
- **Monitoring & Reporting**: Provide comprehensive logging and metrics for system health
|
||||
|
||||
## Architecture
|
||||
|
||||
The system is built around three main worker queues, each handling specific types of workflows:
|
||||
|
||||
### 1. Orchestrator Queue (`orchestrator-queue`)
|
||||
Handles pipeline orchestration and resource management workflows.
|
||||
|
||||
### 2. Alerts Queue (`alerts-queue`)
|
||||
Manages real-time alert notifications and error reporting.
|
||||
|
||||
### 3. Reports Queue (`reports-queue`)
|
||||
Handles scheduled reports and data summaries.
|
||||
|
||||
## Workflows
|
||||
|
||||
### Main Workflows
|
||||
|
||||
#### 1. Orchestrator Workflow
|
||||
**Purpose**: Main orchestration workflow that manages pipeline deployment and resource allocation.
|
||||
|
||||
**Input Parameters**:
|
||||
- `schedule_name` (str): Name of the orchestration schedule
|
||||
- `pipelines_query` (dict): MongoDB query to retrieve pipeline configurations
|
||||
- `opc_servers_query` (dict): MongoDB query to retrieve OPC server configurations
|
||||
|
||||
**What it does**:
|
||||
- Retrieves pipeline configurations from MongoDB
|
||||
- Loads current OPC server slots and active ingestors from Redis
|
||||
- Processes schedules and creates slot configurations
|
||||
- Deploys schedules to Temporal server (scouter and laborious namespaces)
|
||||
- Updates OPC slots in Redis
|
||||
- Generates orchestration reports
|
||||
|
||||
#### 2. Alerts Workflow
|
||||
**Purpose**: Sends real-time error alerts to configured user groups.
|
||||
|
||||
**Input Parameters**:
|
||||
- `schedule_name` (str): Name of the alert schedule
|
||||
- `notification_ttl` (int): Time period before considering notifications persistent
|
||||
- `sent_ttl` (int): Time to live for sent notification cache
|
||||
|
||||
**What it does**:
|
||||
- Filters notifications by ERROR level
|
||||
- Loads notification packages from MongoDB
|
||||
- Applies user group filtering and notification TTL rules
|
||||
- Sends HTML email alerts
|
||||
- Stores notification logs in PostgreSQL
|
||||
- Caches sent notifications to prevent duplicates
|
||||
|
||||
#### 3. Reports Workflow
|
||||
**Purpose**: Sends scheduled reports to configured user groups.
|
||||
|
||||
**Input Parameters**:
|
||||
- `schedule_name` (str): Name of the report schedule
|
||||
|
||||
**What it does**:
|
||||
- Loads all notifications (any level) from MongoDB
|
||||
- Applies user group filtering
|
||||
- Generates HTML report emails
|
||||
- Stores report logs in PostgreSQL
|
||||
|
||||
### Subworkflows
|
||||
|
||||
#### 1. Load Notification Package
|
||||
**Purpose**: Loads notification data and configuration from various sources.
|
||||
|
||||
**Input Parameters**:
|
||||
- `metadata` (dict): Workflow metadata
|
||||
- `mail_type` (str): Type of mail (Alerts/Reports)
|
||||
- `base_data_filter` (dict): Base filters for data retrieval
|
||||
|
||||
**Returns**:
|
||||
- `last_timestamp` (str): Last processed timestamp
|
||||
- `notification_package` (list): Package of notifications to process
|
||||
- `sending_configs` (list): Email sending configurations
|
||||
|
||||
#### 2. Process Notifications
|
||||
**Purpose**: Processes notifications and sends emails with logging.
|
||||
|
||||
**Input Parameters**:
|
||||
- `metadata` (dict): Workflow metadata
|
||||
- `mail_type` (str): Type of mail being sent
|
||||
- `schema` (str): Database schema name
|
||||
- `table_name` (str): Database table name
|
||||
- `notification_package` (list): Notifications to process
|
||||
|
||||
**Returns**:
|
||||
- `log_report` (dict): Report of processed notifications
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Database Connections
|
||||
|
||||
#### Redis Configuration
|
||||
- `REDIS_HOST`: Redis server hostname (default: localhost)
|
||||
- `REDIS_PORT`: Redis server port (default: 6379)
|
||||
- `REDIS_USERNAME`: Redis username (default: default)
|
||||
- `REDIS_PASSWORD`: Redis password (from secret)
|
||||
|
||||
#### MongoDB Configuration
|
||||
- `MONGODB_USERNAME`: MongoDB username (default: root)
|
||||
- `MONGODB_PASSWORD`: MongoDB password
|
||||
- `MONGODB_URL`: MongoDB server URL (default: localhost:27017)
|
||||
- `MONGODB_DATABASE`: Database name (default: sientia)
|
||||
- `MONGODB_TTL_INDEX_HOURS`: TTL index duration in hours (default: 1)
|
||||
|
||||
#### PostgreSQL Configuration
|
||||
- `POSTGRES_HOST`: PostgreSQL server hostname
|
||||
- `POSTGRES_PORT`: PostgreSQL server port (default: 5432)
|
||||
- `POSTGRES_USER`: Database username (default: sientia)
|
||||
- `POSTGRES_PASSWORD`: Database password (default: sientia)
|
||||
- `POSTGRES_DBNAME`: Database name (default: sientia)
|
||||
- `POSTGRES_MIN_CONNECTIONS`: Minimum connection pool size (default: 10)
|
||||
- `POSTGRES_MAX_CONNECTIONS`: Maximum connection pool size (default: 40)
|
||||
|
||||
#### Couchbase Configuration
|
||||
- `COUCHBASE_CONNECTION_STRING`: Couchbase server connection string
|
||||
- `COUCHBASE_USERNAME`: Couchbase username (default: sientia)
|
||||
- `COUCHBASE_PASSWORD`: Couchbase password (default: sientia)
|
||||
|
||||
### Email Configuration
|
||||
- `EMAIL_SENDER`: Sender email address
|
||||
- `EMAIL_SENDER_PASSWORD`: App password for SMTP authentication
|
||||
- `EMAIL_SMTP_SERVER`: SMTP server hostname (default: smtp.gmail.com)
|
||||
- `EMAIL_SMTP_PORT`: SMTP server port (default: 587)
|
||||
|
||||
### Temporal Configuration
|
||||
- `TEMPORAL_HOST`: Temporal server hostname and port
|
||||
- `TEMPORAL_NAMESPACE`: Default Temporal namespace (default: default)
|
||||
- `TEMPORAL_SCOUTER_NAMESPACE`: Scouter workflow namespace (default: scouter)
|
||||
- `TEMPORAL_LABORIOUS_NAMESPACE`: Laborious workflow namespace (default: laborious)
|
||||
|
||||
### Application Configuration
|
||||
- `LOG_LEVEL`: Logging level (default: DEBUG)
|
||||
- `HTTP_METRICS_PORT`: Prometheus metrics port (default: 9090)
|
||||
- `PROJECT_NAME`: Project identifier (default: sientia-orchestrator)
|
||||
- `POD_ID`: Kubernetes pod identifier for metrics
|
||||
|
||||
### Kafka Configuration
|
||||
- `KAFKA_BOOTSTRAP_SERVERS`: Kafka bootstrap servers
|
||||
|
||||
## Deployment
|
||||
|
||||
The system is designed for Kubernetes deployment using Helm charts with:
|
||||
- Health checks and readiness probes
|
||||
- Prometheus metrics endpoint
|
||||
- ServiceMonitor integration for Prometheus Operator
|
||||
- Configurable resource limits and scaling
|
||||
- SSH key management for Git operations
|
||||
|
||||
## Usage
|
||||
|
||||
Workflows can be triggered via Temporal client calls with appropriate input parameters. The system automatically handles:
|
||||
- Pipeline configuration retrieval
|
||||
- Resource allocation
|
||||
- Schedule deployment
|
||||
- Notification processing
|
||||
- Email delivery
|
||||
- Logging and monitoring
|
||||
|
||||
## Monitoring
|
||||
|
||||
The system exposes Prometheus metrics at `/metrics` endpoint including:
|
||||
- Application status (up/down)
|
||||
- Email sent counts
|
||||
- Workflow execution metrics
|
||||
- Custom business metrics
|
||||
|
||||
All operations are logged with structured metadata for debugging and auditing purposes.
|
||||
|
||||
# PR shortcut
|
||||
```
|
||||
git log origin/main..HEAD --no-merges > git_log
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user