# SIENTIA DataOps Orchestrator Temporal

A high-performance, scalable workflow orchestration system built on Temporal.io for automated pipeline management, notification delivery, and resource coordination. The Orchestrator provides enterprise-grade workflow automation, real-time alerting, and comprehensive monitoring capabilities for the SIENTIA platform.

## Features

### Core Functionality
- **Pipeline Orchestration**: Automated deployment and management of data processing pipelines
- **Real-time Notifications**: Intelligent alert filtering and delivery with TTL management
- **Resource Management**: Dynamic OPC server slot allocation and active ingestor monitoring
- **Schedule Management**: Temporal-based workflow scheduling with automatic retry policies
- **Multi-namespace Support**: Separate workflow queues for scouter and laborious operations

### Advanced Capabilities
- **Incremental Processing**: Timestamp-based data loading to avoid reprocessing
- **Configurable Filtering**: User group-based notification filtering with custom policies
- **Auto-scaling Workers**: Multiple worker instances with task queue isolation
- **Comprehensive Logging**: Structured logging with PostgreSQL audit trails
- **Prometheus Metrics**: Real-time monitoring and alerting integration

## Architecture

The SIENTIA DataOps Orchestrator uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in data pipeline environments.

### System Overview

```
┌───────────────────────────────────────────────────────────────────────────────┐
│                              Temporal Cluster                                 │
│  ┌─────────────────┐    ┌──────────────────┐    ┌─────────────────────────┐   │
│  │   Main Worker   │    │  Temporal Client │    │      Task Queues        │   │
│  │                 │◄──►│                  │◄──►│                         │   │
│  │ - Metrics Server│    │ - Namespace Mgmt │    │ - orchestrator-queue    │   │
│  │ - Notifications │    │ - Runtime Config │    │ - alerts-queue          │   │
│  │ - Lifecycle     │    │ - Connection     │    │ - reports-queue         │   │
│  │ - Health Checks │    │ - Security       │    │ - Auto-scaling          │   │
│  └─────────────────┘    └──────────────────┘    └─────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────────────┘
                                        │
                                        ▼
                       ┌────────────────────────────────────────────────────────┐
                       │                    Workflow Layer                      │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐  │
                       │  │   Orchestrator  │  │       Subworkflows          │  │
                       │  │                 │  │                             │  │
                       │  │ - Pipeline Mgmt │  │ - LoadNotificationPackage   │  │
                       │  │ - Resource Mgmt │  │ - ProcessNotifications      │  │
                       │  │ - Schedule Mgmt │  │ - Error Handling            │  │
                       │  └─────────────────┘  └─────────────────────────────┘  │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐  │
                       │  │     Alerts      │  │        Reports              │  │
                       │  │                 │  │                             │  │
                       │  │ - Error Alerts  │  │ - Scheduled Reports         │  │
                       │  │ - TTL Filtering │  │ - Summary Generation        │  │
                       │  │ - Group Filtering│  │ - Comprehensive Logging    │  │
                       │  └─────────────────┘  └─────────────────────────────┘  │
                       └────────────────────────────────────────────────────────┘
                                        │
                                        ▼
                       ┌────────────────────────────────────────────────────────┐
                       │                   Activity Layer                       │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐  │
                       │  │ Temporal Manager│  │      MongoDB Operations     │  │
                       │  │                 │  │                             │  │
                       │  │ - Schedule CRUD │  │ - Document Queries          │  │
                       │  │ - Multi-namespace│  │ - Aggregation Pipelines    │  │
                       │  │ - Normalization │  │ - Timestamp Management      │  │
                       │  │ - Error Handling│  │ - TTL Collections           │  │
                       │  └─────────────────┘  └─────────────────────────────┘  │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐  │
                       │  │  Slot Manager   │  │      Email Services         │  │
                       │  │                 │  │                             │  │
                       │  │ - OPC Slots     │  │ - HTML Generation           │  │
                       │  │ - Active Ingest │  │ - SMTP Management           │  │
                       │  │ - Redis Ops     │  │ - Attachment Handling       │  │
                       │  │ - Cache Mgmt    │  │ - Auto Reconnection         │  │
                       │  └─────────────────┘  └─────────────────────────────┘  │
                       └────────────────────────────────────────────────────────┘
                                        │
                                        ▼
                       ┌────────────────────────────────────────────────────────┐
                       │                  Data Services                         │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐  │
                       │  │     Redis       │  │        MongoDB              │  │
                       │  │                 │  │                             │  │
                       │  │ - OPC Slots     │  │ - Pipeline Configs          │  │
                       │  │ - Timestamps    │  │ - Notification Queue        │  │
                       │  │ - Notification  │  │ - Receiver Groups           │  │
                       │  │ - Cache TTL     │  │ - Orchestrated Schedules    │  │
                       │  └─────────────────┘  └─────────────────────────────┘  │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐  │
                       │  │   PostgreSQL    │  │      SMTP Services          │  │
                       │  │                 │  │                             │  │
                       │  │ - Log Reports   │  │ - Email Delivery            │  │
                       │  │ - Audit Trails  │  │ - Group Management          │  │
                       │  │ - Metrics Data  │  │ - Attachment Support        │  │
                       │  │ - Data Export   │  │ - Security & Auth           │  │
                       │  └─────────────────┘  └─────────────────────────────┘  │
                       └────────────────────────────────────────────────────────┘
                                        │
                                        ▼
                       ┌─────────────────────────────────────────────────────────┐
                       │                External Systems                         │
                       │  ┌─────────────────┐  ┌─────────────────────────────┐   │
                       │  │   Prometheus    │  │      Kubernetes             │   │
                       │  │                 │  │                             │   │
                       │  │ - App Metrics   │  │ - Container Orchestration   │   │
                       │  │ - Email Metrics │  │ - Health Checks             │   │
                       │  │ - SDK Metrics   │  │ - Auto-scaling              │   │
                       │  │ - Alerting      │  │ - Resource Management       │   │
                       │  └─────────────────┘  └─────────────────────────────┘   │
                       └─────────────────────────────────────────────────────────┘
```

### Architecture Principles

#### 1. **Separation of Concerns**
- **Worker Layer**: Manages Temporal workers, task queues, and application lifecycle
- **Workflow Layer**: Orchestrates business logic and process coordination
- **Activity Layer**: Implements specific operations and external system interactions
- **Data Layer**: Handles data persistence, caching, and external service connections

#### 2. **Task Queue Isolation**
- **Orchestrator Queue**: Pipeline and resource management workflows
- **Alerts Queue**: Real-time error notification workflows
- **Reports Queue**: Scheduled reporting and summary workflows

#### 3. **Fault Tolerance & Resilience**
- **Automatic Retry Policies**: Configurable retry strategies for transient failures
- **Graceful Degradation**: System continues operating with reduced functionality
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
- **Connection Management**: Automatic reconnection for SMTP and database services

#### 4. **Scalability & Performance**
- **Horizontal Scaling**: Multiple worker instances for load distribution
- **Connection Pooling**: Optimized database and Redis connections
- **Asynchronous Processing**: Non-blocking operations for improved throughput
- **Resource Optimization**: Intelligent slot allocation and ingestor management

## 🔄 Workflows

### Main Workflows

#### 1. Orchestrator Workflow (`orchestrator.py`)

The **Orchestrator** workflow is the main coordination workflow that manages pipeline deployment and resource allocation across the SIENTIA platform.

**Purpose**:
- **Pipeline Management**: Coordinates deployment of scouter and laborious pipelines
- **Resource Allocation**: Manages OPC server slots and active ingestor distribution
- **Schedule Synchronization**: Ensures Temporal schedules match MongoDB configurations
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules

**Execution Flow**:
1. **Configuration Loading**: Retrieves pipeline and OPC server configurations from MongoDB
2. **Resource Assessment**: Loads current OPC slots and active ingestors from Redis
3. **Schedule Processing**: Formats configurations for different workflow types
4. **Deployment Operations**: Creates, updates, or deletes Temporal schedules
5. **Resource Updates**: Updates OPC slots and MongoDB timestamps
6. **Reporting**: Generates comprehensive orchestration reports

**Input Parameters**:
```json
{
  "schedule_name": "hourly_orchestration",
  "pipelines_query": {
    "collection": "pipelines",
    "aggregation": [
      {"$match": {"active": true}},
      {"$sort": {"updated_at": -1}}
    ]
  },
  "opc_servers_query": {
    "collection": "opc_servers",
    "filters": {"active": true}
  }
}
```

#### 2. Alerts Workflow (`alerts.py`)

The **Alerts** workflow processes and sends real-time error notifications to configured user groups with intelligent filtering and duplicate prevention.

**Purpose**:
- **Error Alerting**: Immediate notification of ERROR-level events
- **TTL Management**: Prevents alert spam using configurable time-to-live settings
- **Group Filtering**: Sends alerts only to relevant user groups
- **Persistent Monitoring**: Tracks and escalates persistent issues

**Execution Flow**:
1. **Notification Loading**: Retrieves ERROR-level notifications from MongoDB
2. **Timestamp Filtering**: Applies incremental processing using Redis timestamps
3. **Group Filtering**: Filters notifications by user group configurations
4. **TTL Processing**: Checks notification cache to prevent duplicate alerts
5. **Email Generation**: Creates HTML email content for each group
6. **Delivery & Logging**: Sends emails and logs results to PostgreSQL

**Input Parameters**:
```json
{
  "schedule_name": "error_alerts",
  "notification_ttl": 3600,
  "sent_ttl": 7200
}
```

#### 3. Reports Workflow (`reports.py`)

The **Reports** workflow generates and sends scheduled comprehensive reports to configured user groups.

**Purpose**:
- **Scheduled Reporting**: Regular summary reports of system activity
- **Comprehensive Coverage**: Includes all notification levels (not just errors)
- **Group Management**: Customizable reports per user group
- **Audit Trail**: Complete logging of report delivery

**Execution Flow**:
1. **Data Collection**: Loads all notifications from MongoDB (any level)
2. **Timestamp Processing**: Uses incremental loading with Redis timestamps
3. **Group Processing**: Applies user group filtering for report customization
4. **Report Generation**: Creates HTML reports with comprehensive summaries
5. **Distribution**: Sends reports to configured recipients
6. **Audit Logging**: Records delivery status in PostgreSQL

### Subworkflows

#### 1. Load Notification Package (`load_notification_package.py`)

**Purpose**: Centralized notification data loading and configuration management for both alerts and reports workflows.

**Key Features**:
- **Incremental Processing**: Uses Redis timestamps for efficient data loading
- **Configuration Management**: Loads active receiver group configurations
- **Data Validation**: Ensures complete data packages before processing
- **Timestamp Management**: Updates last processed timestamps

**Input Parameters**:
```json
{
  "metadata": {"workflow_name": "alerts", "schedule_name": "error_alerts"},
  "mail_type": "Alerts",
  "base_data_filter": {"level": "ERROR"}
}
```

**Returns**:
- `last_timestamp` (str | None): Last processed timestamp
- `notification_package` (list[dict]): Retrieved notifications
- `sending_configs` (list[dict]): Active receiver group configurations

#### 2. Process Notifications (`process_notifications.py`)

**Purpose**: Handles email generation, delivery, and audit logging for notification workflows.

**Key Features**:
- **HTML Generation**: Creates formatted email content for each receiver group
- **Email Delivery**: Sends emails with attachment support and error handling
- **Audit Logging**: Records delivery status and metrics in PostgreSQL
- **Error Recovery**: Handles SMTP failures with detailed error reporting

**Input Parameters**:
```json
{
  "metadata": {"workflow_name": "alerts", "schedule_name": "error_alerts"},
  "mail_type": "Alerts",
  "schema": "sientia_data",
  "table_name": "log_report",
  "notification_package": {
    "group_name": "admin_team",
    "members": ["admin@example.com"],
    "notifications": [...]
  }
}
```

**Returns**:
- `log_report` (list[dict]): Detailed delivery status for each notification

### Key Components

#### **Worker (`orchestrator/worker/worker.py`)**
- **Purpose**: Main application orchestrator managing Temporal workers and task queues
- **Responsibilities**:
  - Temporal client initialization and connection management
  - Worker lifecycle management and graceful shutdown
  - Task queue configuration (orchestrator, alerts, reports)
  - Prometheus metrics server initialization
  - Notification handler setup and configuration
- **Key Features**:
  - Multi-queue worker management with automatic scaling
  - Health check endpoints for Kubernetes liveness/readiness probes
  - Graceful shutdown with cleanup procedures
  - Comprehensive error handling and metrics collection

#### **Activities (`orchestrator/activities/`)**
- **Activities**: Main activity orchestrator combining all operations
- **TemporalManager**: Temporal schedule CRUD operations across namespaces
- **SlotManager**: Redis-based OPC slot and cache management
- **MongoDB**: Document operations, aggregations, and TTL management
- **Email**: SMTP operations with HTML generation and attachment support
- **Formatters**: Configuration processing and slot distribution algorithms

#### **Utilities (`orchestrator/utils/`)**
- **Connectors Configuration**: Database and service configuration management
- **Email Builder**: HTML email template generation and formatting
- **Orchestrator Functions**: Pipeline configuration transformation utilities
- **Converters**: Data type conversion and validation utilities

## 📋 Prerequisites

- Python 3.11+
- Temporal server/cluster
- Redis server
- MongoDB server
- PostgreSQL database
- SMTP server access

**Note**: External dependencies must be available either through:
- Kubernetes cluster deployment
- Docker Compose setup
- Cloud-managed services
- Local installations

## 🚀 Installation

### Local Development Setup

1. **Clone the repository**
   ```bash
   git clone <repository-url>
   cd sientia-dataops-orchestrator_temporal
   ```

2. **Create virtual environment**
   ```bash
   python3.11 -m venv venv
   source ./venv/bin/activate
   ```

3. **Install dependencies**
   ```bash
   pip install -r requirements.txt
   ```

4. **Configure environment variables**
   ```bash
   # Set required environment variables for services
   export TEMPORAL_HOST=localhost:7233
   export REDIS_HOST=localhost
   export MONGODB_URL=localhost:27017
   export POSTGRES_HOST=localhost
   # ... additional configuration
   ```

## 📦 How to Run

### Running the Orchestrator Application

Use the provided script to run the application locally:

```bash
# Make script executable (first time only)
chmod +x run_local.sh

# Run the application
./run_local.sh
```

The script will:
- Activate the virtual environment
- Load environment variables from `.env`
- Start the orchestrator worker application

### Running Tests and Coverage

Use the provided script to run tests with coverage:

```bash
# Make script executable (first time only)
chmod +x run_coverage.sh

# Run tests with coverage
./run_coverage.sh
```

### Manual Application Execution

For manual execution without scripts:

```bash
# Activate virtual environment
source ./venv/bin/activate

# Start the orchestrator worker
python -m orchestrator.worker.worker
```

## ⚙️ Configuration

### Environment Variables

| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
| `TEMPORAL_NAMESPACE` | Default Temporal namespace | `default` | No |
| `TEMPORAL_SCOUTER_NAMESPACE` | Scouter workflow namespace | `scouter` | No |
| `TEMPORAL_LABORIOUS_NAMESPACE` | Laborious workflow namespace | `laborious` | No |
| `REDIS_HOST` | Redis server hostname | `localhost` | Yes |
| `REDIS_PORT` | Redis server port | `6379` | Yes |
| `REDIS_USERNAME` | Redis username | `default` | Yes |
| `REDIS_PASSWORD` | Redis password | - | Yes |
| `MONGODB_URL` | MongoDB server URL | `localhost:27017` | Yes |
| `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
| `MONGODB_PASSWORD` | MongoDB password | - | Yes |
| `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes |
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
| `POSTGRES_PASSWORD` | PostgreSQL password | - | Yes |
| `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
| `EMAIL_SENDER` | Sender email address | - | Yes |
| `EMAIL_SENDER_PASSWORD` | SMTP password | - | Yes |
| `EMAIL_SMTP_SERVER` | SMTP server | `smtp.gmail.com` | No |
| `EMAIL_SMTP_PORT` | SMTP port | `587` | No |
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |

### Workflow Configuration

Temporal input configuration sample:

#### Orchestrator Workflow

```json
{
    "schedule_name": "orchestrator-test",
    "pipelines_query": {
        "collection": "pipelines",
        "aggregation": [
            {
                "$lookup": {
                    "from": "models",
                    "localField": "model_id",
                    "foreignField": "id",
                    "as": "model_docs"
                }
            },
            {
                "$match": {
                    "active": True
                }
            },
            {
                "$addFields": {
                    "models": {
                        "$arrayElemAt": [
                            "$model_docs",
                            0
                        ]
                    }
                }
            },
            {
                "$match": {
                    "models.active": True
                }
            },
            {
                "$project": {
                    "model_docs": 0
                }
            }
        ]
    },
    "opc_servers_query": {
        "collection": "opc-servers",
        "filters": {

        }
    }
}
```

#### Alerts Workflow

```json
{
    "schedule_name": "alerts",
    "notification_ttl": 5*60,
    "sent_ttl": 10*60
}
```

#### Reports Workflow

```json
{
    "schedule_name": "reports",
}
```

## 📊 Monitoring and Metrics

The Orchestrator system exposes comprehensive Prometheus metrics:

### Application Metrics
- `app_up`: Application health status (1=healthy, 0=unhealthy)
- `email_sent_count`: Email delivery operation count by group

### Workflow Metrics
- Schedule creation, update, and deletion success rates
- Notification processing times and error rates
- Resource allocation and slot management metrics

### Database Metrics
- MongoDB query performance and connection health
- Redis operation counts and response times
- PostgreSQL export operations and audit log metrics

## 🧪 Testing

### Test Structure
```
tests/
├── orchestrator/           # Orchestrator workflow tests
├── activities/            # Activity implementation tests
├── utils/                 # Utility function tests
└── integration/           # End-to-end workflow tests
```

### Test Execution
```bash
# Install test dependencies
pip install pytest pytest-cov pytest-asyncio

# Run tests with coverage
pytest --cov=orchestrator --cov-report=html

# Run specific test modules
pytest tests/activities/test_mongo_db.py
pytest tests/workflows/test_orchestrator.py
```

## 🔧 Development

### Project Structure
```
orchestrator/
├── activities/              # Temporal activity implementations
│   ├── activities.py       # Main activities orchestrator
│   ├── temporal_manager.py # Temporal schedule operations
│   ├── slot_manager.py     # Redis slot management
│   ├── mongo_db.py         # MongoDB operations
│   ├── email.py            # Email service operations
│   └── formatters.py       # Configuration formatting
├── workflows/               # Temporal workflow definitions
│   ├── orchestrator.py     # Main orchestration workflow
│   ├── alerts.py           # Error alert workflow
│   ├── reports.py          # Scheduled report workflow
│   └── subworkflows/       # Sub-workflow implementations
├── worker/                  # Worker implementation
│   └── worker.py           # Main worker orchestrator
├── utils/                   # Utility functions
│   ├── connectors_config.py # Database configuration
│   ├── email_builder.py    # Email template generation
│   └── orchestrator_functions.py # Pipeline utilities
└── metrics.py               # Prometheus metrics definitions
```

### Adding New Features

1. **Follow Temporal patterns** for new workflows and activities
2. **Add comprehensive docstrings** for all public methods
3. **Include Prometheus metrics** for monitoring
4. **Add unit tests** for new functionality
5. **Update this README** with new features and configuration

## 🐛 Troubleshooting

### Common Issues

1. **Temporal Connection Failures**
   - Verify Temporal server is running and accessible
   - Check namespace configuration and permissions
   - Review server logs for connection issues

2. **Database Connection Issues**
   - Verify all database services are running
   - Check connection credentials and network access
   - Ensure proper connection pool configuration

3. **Email Delivery Failures**
   - Verify SMTP server configuration and credentials
   - Check email sender permissions and authentication
   - Review email delivery logs for specific errors

4. **Workflow Execution Failures**
   - Review activity error logs and notifications
   - Check MongoDB collection configurations
   - Verify input data format and required fields

### Debug Mode

Enable debug logging by setting the log level:
```bash
export LOG_LEVEL=DEBUG
```

## ⚡ Performance Tuning

### Key Parameters

- **Worker Concurrency**: Configure worker task limits in Temporal client
- **Connection Pools**: Optimize database connection pool sizes
- **Redis TTL**: Adjust cache TTL settings based on requirements
- **Batch Sizes**: Configure notification processing batch sizes

### Scaling Considerations

- **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Use dedicated queues for different workflows
- **Database Performance**: Optimize indexes and connection pooling
- **Memory Management**: Monitor and configure appropriate resource limits

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes with comprehensive testing
4. Update documentation and docstrings
5. Submit a pull request

### Code Quality Standards

- Follow PEP 8 style guidelines
- Include comprehensive docstrings for all public methods
- Maintain test coverage above 80%
- Use type hints where appropriate
- Follow Temporal.io best practices

## 📄 License

This project is licensed under the terms specified in the LICENSE file.

## 🆘 Support

For support and questions:
- Check the troubleshooting section above
- Review the metrics and logs for error patterns
- Open an issue in the project repository
- Contact the development team

---

**Note**: The SIENTIA DataOps Orchestrator is designed for production use in enterprise data environments. Ensure proper security configuration and network isolation for production deployments.