SIENTIA DataOps Orchestrator Temporal

Overview

The SIENTIA DataOps Orchestrator Temporal is a comprehensive workflow orchestration system built on Temporal.io that manages data pipelines, notifications, and system orchestration for the SIENTIA platform. It provides automated scheduling, monitoring, and execution of data processing workflows with integrated alerting and reporting capabilities.

Project Goals

• Pipeline Orchestration: Automate the deployment and management of data processing pipelines
• Notification Management: Handle real-time alerts and scheduled reports for system events
• Resource Management: Manage OPC server slots and data ingestion resources
• Workflow Automation: Coordinate complex workflows across multiple services and databases
• Monitoring & Reporting: Provide comprehensive logging and metrics for system health

Architecture

The system is built around three main worker queues, each handling specific types of workflows:

1. Orchestrator Queue (orchestrator-queue)

Handles pipeline orchestration and resource management workflows.

2. Alerts Queue (alerts-queue)

Manages real-time alert notifications and error reporting.

3. Reports Queue (reports-queue)

Handles scheduled reports and data summaries.

Workflows

Main Workflows

1. Orchestrator Workflow

Purpose: Main orchestration workflow that manages pipeline deployment and resource allocation.

Input Parameters:

• schedule_name (str): Name of the orchestration schedule
• pipelines_query (dict): MongoDB query to retrieve pipeline configurations
• opc_servers_query (dict): MongoDB query to retrieve OPC server configurations

What it does:

• Retrieves pipeline configurations from MongoDB
• Loads current OPC server slots and active ingestors from Redis
• Processes schedules and creates slot configurations
• Deploys schedules to Temporal server (scouter and laborious namespaces)
• Updates OPC slots in Redis
• Generates orchestration reports

2. Alerts Workflow

Purpose: Sends real-time error alerts to configured user groups.

Input Parameters:

• schedule_name (str): Name of the alert schedule
• notification_ttl (int): Time period before considering notifications persistent
• sent_ttl (int): Time to live for sent notification cache

What it does:

• Filters notifications by ERROR level
• Loads notification packages from MongoDB
• Applies user group filtering and notification TTL rules
• Sends HTML email alerts
• Stores notification logs in PostgreSQL
• Caches sent notifications to prevent duplicates

3. Reports Workflow

Purpose: Sends scheduled reports to configured user groups.

Input Parameters:

• schedule_name (str): Name of the report schedule

What it does:

• Loads all notifications (any level) from MongoDB
• Applies user group filtering
• Generates HTML report emails
• Stores report logs in PostgreSQL

Subworkflows

1. Load Notification Package

Purpose: Loads notification data and configuration from various sources.

Input Parameters:

• metadata (dict): Workflow metadata
• mail_type (str): Type of mail (Alerts/Reports)
• base_data_filter (dict): Base filters for data retrieval

Returns:

• last_timestamp (str): Last processed timestamp
• notification_package (list): Package of notifications to process
• sending_configs (list): Email sending configurations

2. Process Notifications

Purpose: Processes notifications and sends emails with logging.

Input Parameters:

• metadata (dict): Workflow metadata
• mail_type (str): Type of mail being sent
• schema (str): Database schema name
• table_name (str): Database table name
• notification_package (list): Notifications to process

Returns:

• log_report (dict): Report of processed notifications

Environment Variables

Database Connections

Redis Configuration

• REDIS_HOST: Redis server hostname (default: localhost)
• REDIS_PORT: Redis server port (default: 6379)
• REDIS_USERNAME: Redis username (default: default)
• REDIS_PASSWORD: Redis password (from secret)

MongoDB Configuration

• MONGODB_USERNAME: MongoDB username (default: root)
• MONGODB_PASSWORD: MongoDB password
• MONGODB_URL: MongoDB server URL (default: localhost:27017)
• MONGODB_DATABASE: Database name (default: sientia)
• MONGODB_TTL_INDEX_HOURS: TTL index duration in hours (default: 1)

PostgreSQL Configuration

• POSTGRES_HOST: PostgreSQL server hostname
• POSTGRES_PORT: PostgreSQL server port (default: 5432)
• POSTGRES_USER: Database username (default: sientia)
• POSTGRES_PASSWORD: Database password (default: sientia)
• POSTGRES_DBNAME: Database name (default: sientia)
• POSTGRES_MIN_CONNECTIONS: Minimum connection pool size (default: 10)
• POSTGRES_MAX_CONNECTIONS: Maximum connection pool size (default: 40)

Couchbase Configuration

• COUCHBASE_CONNECTION_STRING: Couchbase server connection string
• COUCHBASE_USERNAME: Couchbase username (default: sientia)
• COUCHBASE_PASSWORD: Couchbase password (default: sientia)

Email Configuration

• EMAIL_SENDER: Sender email address
• EMAIL_SENDER_PASSWORD: App password for SMTP authentication
• EMAIL_SMTP_SERVER: SMTP server hostname (default: smtp.gmail.com)
• EMAIL_SMTP_PORT: SMTP server port (default: 587)

Temporal Configuration

• TEMPORAL_HOST: Temporal server hostname and port
• TEMPORAL_NAMESPACE: Default Temporal namespace (default: default)
• TEMPORAL_SCOUTER_NAMESPACE: Scouter workflow namespace (default: scouter)
• TEMPORAL_LABORIOUS_NAMESPACE: Laborious workflow namespace (default: laborious)

Application Configuration

• LOG_LEVEL: Logging level (default: DEBUG)
• HTTP_METRICS_PORT: Prometheus metrics port (default: 9090)
• PROJECT_NAME: Project identifier (default: sientia-orchestrator)
• POD_ID: Kubernetes pod identifier for metrics

Kafka Configuration

• KAFKA_BOOTSTRAP_SERVERS: Kafka bootstrap servers

Deployment

The system is designed for Kubernetes deployment using Helm charts with:

• Health checks and readiness probes
• Prometheus metrics endpoint
• ServiceMonitor integration for Prometheus Operator
• Configurable resource limits and scaling
• SSH key management for Git operations

Usage

Workflows can be triggered via Temporal client calls with appropriate input parameters. The system automatically handles:

• Pipeline configuration retrieval
• Resource allocation
• Schedule deployment
• Notification processing
• Email delivery
• Logging and monitoring

Monitoring

The system exposes Prometheus metrics at /metrics endpoint including:

• Application status (up/down)
• Email sent counts
• Workflow execution metrics
• Custom business metrics

All operations are logged with structured metadata for debugging and auditing purposes.

PR shortcut

git log origin/main..HEAD --no-merges > git_log

Prompt: Write a summary of PR changes in markdown. Be objective and direct. Write to file