vitor-aignosi
7e6f1a8782
Resolve PI Web API pipeline tags through the tag database - Load active `pi_web_api` connectors and referenced tags behind the `connectors-pi-web-api` patch, so replays of older runs are unaffected. - Resolve `pi_web_api_scouter` read tags and `predictions_batch` output tags by `tag_id`; legacy `webid` pipelines produce identical input. - Add a `connection` block to scouter schedules with the secret reference only, plus connector defaults for period, max_count and api_timeout. - Hold pipelines whose tags span several connectors and notify unresolved tags, wrong categories and missing connectors once per run. - Share the issue notification logic between slots and schedules.
1070 lines
46 KiB
Markdown
1070 lines
46 KiB
Markdown
# SIENTIA DataOps Orchestrator Temporal
|
|
|
|
A high-performance, scalable workflow orchestration system built on Temporal.io for automated pipeline management, notification delivery, and resource coordination. The Orchestrator provides enterprise-grade workflow automation, real-time alerting, and comprehensive monitoring capabilities for the SIENTIA platform.
|
|
|
|
## 📑 Table of Contents
|
|
|
|
- [Features](#features)
|
|
- [Core Functionality](#core-functionality)
|
|
- [Advanced Capabilities](#advanced-capabilities)
|
|
- [Architecture](#architecture)
|
|
- [Architecture Principles](#architecture-principles)
|
|
- [Task Queue Isolation](#2-task-queue-isolation)
|
|
- [Workflows](#-workflows)
|
|
- [Orchestrator Workflow](#1-orchestrator-workflow-orchestratorpy)
|
|
- [Alerts Workflow](#2-alerts-workflow-alertspy)
|
|
- [Reports Workflow](#3-reports-workflow-reportspy)
|
|
- [Subworkflows](#subworkflows)
|
|
- [Load Notification Package](#1-load-notification-package-load_notification_packagepy)
|
|
- [Process Notifications](#2-process-notifications-process_notificationspy)
|
|
- [Notification Filtering System](#-notification-filtering-system)
|
|
- [Prerequisites](#-prerequisites)
|
|
- [Installation](#-installation)
|
|
- [How to Run](#-how-to-run)
|
|
- [Configuration](#-configuration)
|
|
- [Monitoring and Metrics](#-monitoring-and-metrics)
|
|
- [Testing](#-testing)
|
|
- [Code Quality & Validation](#-code-quality--validation)
|
|
- [Development](#-development)
|
|
- [Troubleshooting](#-troubleshooting)
|
|
- [Performance Tuning](#-performance-tuning)
|
|
- [Contributing](#-contributing)
|
|
- [License](#-license)
|
|
- [Support](#-support)
|
|
|
|
## Features
|
|
|
|
### Core Functionality
|
|
- **Pipeline Orchestration**: Automated deployment and management of data processing pipelines
|
|
- **Real-time Notifications**: Intelligent alert filtering and delivery with TTL management
|
|
- **Resource Management**: Dynamic OPC server slot allocation and active ingestor monitoring
|
|
- **Schedule Management**: Temporal-based workflow scheduling with automatic retry policies
|
|
- **Multi-namespace Support**: Separate workflow queues for scouter and laborious operations
|
|
|
|
### Advanced Capabilities
|
|
- **Incremental Processing**: Timestamp-based data loading to avoid reprocessing
|
|
- **Intelligent Notification Filtering**: Advanced filtering system with:
|
|
- User group-based notification filtering with custom policies
|
|
- TTL-based duplicate prevention for alerts
|
|
- Ignore lists for specific notifications
|
|
- Persistent alert detection for ongoing issues
|
|
- **Auto-scaling Workers**: Multiple worker instances with task queue isolation
|
|
- **Comprehensive Logging**: Structured logging with PostgreSQL audit trails
|
|
- **Prometheus Metrics**: Real-time monitoring and alerting integration
|
|
- **Template-based Email Generation**: Jinja2-powered HTML email templates
|
|
|
|
## Architecture
|
|
|
|
The SIENTIA DataOps Orchestrator uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in data pipeline environments.
|
|
|
|
### Architecture Principles
|
|
|
|
#### 1. **Separation of Concerns**
|
|
- **Worker Layer**: Manages Temporal workers, task queues, Prometheus metrics, and application lifecycle
|
|
- **Workflow Layer**: Orchestrates business logic and process coordination with subworkflows for modularity
|
|
- **Activity Layer**: Implements specific operations and external system interactions with error handling
|
|
- **Data Layer**: Handles data persistence (MongoDB, PostgreSQL), caching (Redis), and external service connections
|
|
|
|
#### 2. **Task Queue Isolation**
|
|
- **Orchestrator Queue**: Pipeline and resource management workflows (orchestrator workflow)
|
|
- **Alerts Queue**: Real-time error notification workflows (alerts workflow, load_notification_package, process_notifications subworkflows)
|
|
- **Reports Queue**: Scheduled reporting and summary workflows (reports workflow, load_notification_package, process_notifications subworkflows)
|
|
|
|
#### 3. **Fault Tolerance & Resilience**
|
|
- **Automatic Retry Policies**: Configurable retry strategies for transient failures
|
|
- **Graceful Degradation**: System continues operating with reduced functionality
|
|
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
|
|
- **Connection Management**: Automatic reconnection for SMTP and database services
|
|
|
|
#### 4. **Scalability & Performance**
|
|
- **Horizontal Scaling**: Multiple worker instances for load distribution
|
|
- **Connection Pooling**: Optimized database and Redis connections
|
|
- **Asynchronous Processing**: Non-blocking operations for improved throughput
|
|
- **Resource Optimization**: Intelligent slot allocation and ingestor management
|
|
|
|
## 🔄 Workflows
|
|
|
|
### Main Workflows
|
|
|
|
#### 1. Orchestrator Workflow (`orchestrator.py`)
|
|
|
|
The **Orchestrator** workflow is the main coordination workflow that manages pipeline deployment and resource allocation across the SIENTIA platform.
|
|
|
|
**Purpose**:
|
|
- **Pipeline Management**: Coordinates deployment of scouter and laborious pipelines
|
|
- **Resource Allocation**: Manages OPC server slots and active ingestor distribution
|
|
- **Schedule Synchronization**: Ensures Temporal schedules match MongoDB configurations
|
|
- **Infrastructure Management**: Creates, updates, and deletes workflow schedules
|
|
|
|
**Execution Flow**:
|
|
1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, active ingestors, and the tag database (`connectors` and `connector_tags`, OPC UA for the slot and PI Web API for the schedules)
|
|
2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots
|
|
3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes
|
|
4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently
|
|
5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines
|
|
|
|
**Input Parameters**:
|
|
```json
|
|
{
|
|
"schedule_name": "hourly_orchestration",
|
|
"pipelines_query": {
|
|
"collection": "pipelines",
|
|
"aggregation": [
|
|
{"$match": {"active": true}},
|
|
{"$sort": {"updated_at": -1}}
|
|
]
|
|
},
|
|
"opc_servers_query": {
|
|
"collection": "opc_servers",
|
|
"filters": {"active": true}
|
|
}
|
|
}
|
|
```
|
|
|
|
Optional: `connectors_query` and `connector_tags_query` override the tag database aggregations. The defaults live in the code (`orchestrator/workflows/orchestrator.py`), so the schedule input does not need them:
|
|
|
|
```json
|
|
{
|
|
"connectors_query": {
|
|
"collection": "connectors",
|
|
"aggregation": [
|
|
{"$match": {"type": "opc_ua", "status": "Active"}},
|
|
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "server_uri": 1,
|
|
"use_security": 1, "subscription_period_ms": 1}}
|
|
]
|
|
},
|
|
"connector_tags_query": {
|
|
"collection": "connector_tags",
|
|
"aggregation": [
|
|
{"$match": {"category": "Read", "status": "Active"}},
|
|
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1}}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
The tag database is read only when a scouter read tag carries a `tag_id`, and the workflow puts `{"$match": {"tag_id": {"$in": [...]}}}` in front of `connector_tags_query`, so it loads only the referenced tags. Environments without a tag database, or whose pipelines do not use `tag_id`, never query it. An override should keep a `$project` of these fields: any other attribute may hold a BSON type a Temporal payload cannot carry (such as `ObjectId`).
|
|
|
|
#### OPC slots (slot v2, contract C1)
|
|
|
|
`process_slots` writes one slot per active ingestor to Redis (`slot:opc_tags:{n}`). The OPC Gateway reads it (SIENTIAPDE-1974). Each server carries its connection fields, and each tag address carries the list of names it is written under:
|
|
|
|
```json
|
|
{
|
|
"kepserver": {
|
|
"server_id": "1", "name": "kepserver", "url": "opc.tcp://host:49320",
|
|
"server_uri": "urn:...", "use_security": true, "subscription_period_ms": 7500.0,
|
|
"tags": {
|
|
"ns=2;s=Channel1.Device1.Tag1": {
|
|
"frequency": 15000,
|
|
"active_read": {"enabled": false},
|
|
"names": [
|
|
{"collection": "raw_model_a", "tag_name": "Counter", "model_id": "1", "model_name": "model a"},
|
|
{"collection": "raw_model_b", "tag_name": "Contador", "model_id": "2", "model_name": "model b"}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
How each scouter `read_tags` entry is resolved:
|
|
|
|
| Read tag | Address and server | Name |
|
|
|---|---|---|
|
|
| With `tag_id` | `connector_tags` (`address`, `connector_id`) and `connectors` (`url`, `server_uri`, `use_security`, `subscription_period_ms`) | The read tag's `tag_name`, else the `connector_tags` one |
|
|
| Without `tag_id` | `server_id` + `tag_address`, server from `opc-servers` | The read tag's `tag_name` |
|
|
|
|
Rules:
|
|
- **One entry per address.** An address read by N pipelines is read once and written to N collections: one `names` item per pipeline (`collection` = `raw_{schedule_name}`).
|
|
- **Same name source as the scouter.** `resolve_tag_names` names both the slot's `names` and the scouter's `model_tags`, so the gateway always writes the name the scouter looks for.
|
|
- **`frequency`:** the lowest among the pipelines of the address, as an integer. Every read tag needs one, with or without `tag_id`.
|
|
- **`tag_id`:** a missing, `null` or empty `tag_id` means the read tag does not use the tag database.
|
|
- **`subscription_period_ms`:** the connector's value. For `opc-servers`, it is half the server's lowest tag frequency in the slot, as before.
|
|
- **`use_security`:** the connector's value. For `opc-servers`, it is `true` when the server has a `cert_path`. The certificate itself comes from MinIO on the gateway, so the slot carries no certificate path.
|
|
- **`active_read`:** off by default. It is on when any pipeline sets `read_tags[].active_read.enabled`, with the lowest `cycle_multiple`.
|
|
- **Same `server_id` in both sources:** a server known through `connectors` and `opc-servers` under the same id is one server, and the `connectors` fields win. Two different servers with the same name get distinct slot keys (`name`, then `name:server_id`).
|
|
- **Slot division:** the addresses are split across the slots, sorted by server and address, and the names of an address always stay in the same slot.
|
|
- **Unresolved tags:** a read tag that cannot be resolved is left out of the slot, and the rest of the slot is still written. It is notified once per run:
|
|
- `ORCHESTRATOR_TAG_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `tag_id` is not an active Read tag, or its document lacks `connector_id`, `address` or `tag_name`;
|
|
- `ORCHESTRATOR_CONNECTOR_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the connector is missing or inactive, or lacks `name`, `url` or `server_uri`;
|
|
- `ORCHESTRATOR_SERVER_NOT_FOUND_DURING_SLOT_CONFIGURATION`: the `server_id` is not in `opc-servers`;
|
|
- `ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION`: the read tag has no integer `frequency` or, without `tag_id`, lacks `server_id`, `tag_address` or `tag_name`.
|
|
- **Only changed slots are written.** The slot is built in a stable order, so unchanged pipelines produce an identical slot, and the gateway does not re-subscribe. A slot still in the v1 format (`topics`, `cert_path`) is replaced once.
|
|
|
|
**Gateway plan B:** the gateway's legacy read engine (`READ_ENGINE=legacy`, kept until the end of SIENTIAPDE-1974) reads the slot v1 only. Once this version writes the slot v2, switching the gateway to the legacy engine also requires rolling the Orchestrator back to a version that writes v1.
|
|
|
|
**Known limitation:** scouter schedules are only redeployed when the pipeline's `updated_at` changes. If a tag is renamed in `connector_tags`, the slot picks up the new name on the next run, but the scouter keeps the old `model_tags` until the pipeline is updated. When renaming, also update the pipeline, or keep the name in `read_tags[].tag_name`.
|
|
|
|
#### PI Web API through the tag database
|
|
|
|
The tag database also holds PI Web API servers: `connectors` documents with `type: "pi_web_api"`, whose `url` is the PI Web API base URL and whose `pi_web_api` block carries `auth` (`type`: `basic` | `bearer`, `secret`: `{name, key}` of a Kubernetes secret, never the credential) and optional connection defaults (`period`, `max_count`, `api_timeout`, `data_server_webid`). Their tags are ordinary `connector_tags` whose `address` is the PI WebID. The slot never reads them: they only feed `process_schedules`.
|
|
|
|
The workflow loads them behind the `connectors-pi-web-api` patch, with two more optional overrides (defaults in `orchestrator/workflows/orchestrator.py`):
|
|
|
|
```json
|
|
{
|
|
"pi_connectors_query": {
|
|
"collection": "connectors",
|
|
"aggregation": [
|
|
{"$match": {"type": "pi_web_api", "status": "Active"}},
|
|
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "pi_web_api": 1}}
|
|
]
|
|
},
|
|
"pi_connector_tags_query": {
|
|
"collection": "connector_tags",
|
|
"aggregation": [
|
|
{"$match": {"status": "Active"}},
|
|
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1, "category": 1}}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
As for OPC, they are read only when a PI Web API tag carries a `tag_id`, and the workflow puts the referenced `tag_id`s in front of `pi_connector_tags_query`. Tags of every category are loaded, since `predictions_batch` writes to `Write` and `Prediction Confidence` tags.
|
|
|
|
How the PI Web API pipelines are resolved:
|
|
|
|
| Pipeline field | With `tag_id` | Without `tag_id` |
|
|
|---|---|---|
|
|
| `pi_web_api_scouter` `read_tags` (`{model_tag: {...}}`) | `webid` = the `Read` tag's `address`; the key, `aggr_func` and `data_range` stay the pipeline's | `webid` from the pipeline, as before |
|
|
| `predictions_batch` `pi_web_api_output_config.prediction_tags` / `confidence_tags` given as a list | `{tag_name: address}` of the `Write` / `Prediction Confidence` tag | A mapping already given as `{tag_name: webid}` passes through unchanged |
|
|
|
|
Rules:
|
|
- **Connection.** When a `pi_web_api_scouter` resolves at least one tag through the tag database, its schedule input gets `connection: {connector_id, base_url, auth: {type, secret: {name, key}}}` (plus `data_server_webid` when set), and the connector's `period`, `max_count` and `api_timeout` become defaults under the pipeline's `pi_web_api_config`: pipeline values win, and `api_timeout` is still capped at the frequency. `connection` is additive: the scouter keeps reading `pi_web_api_query` until its own card. Only the secret reference is copied; the orchestrator never resolves the secret.
|
|
- **Legacy pipelines are unchanged.** Without any `tag_id`, the schedule input is byte-identical to before.
|
|
- **One connector per pipeline.** A `pi_web_api_scouter` whose tags belong to two connectors is held: its existing schedule is neither updated nor deleted (and none is created), and the conflict is notified.
|
|
- **Unresolved tags** are left out, the rest of the pipeline is still scheduled, and they are notified once per run (values are `schedule_name/tag_id`):
|
|
- `ORCHESTRATOR_PI_TAG_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the `tag_id` is not an active tag, or its document lacks `connector_id`, `address` or `tag_name`;
|
|
- `ORCHESTRATOR_PI_TAG_WRONG_CATEGORY_DURING_SCHEDULE_CONFIGURATION`: the tag's category does not fit (`Read` for read tags, `Write` for `prediction_tags`, `Prediction Confidence` for `confidence_tags`);
|
|
- `ORCHESTRATOR_PI_CONNECTOR_NOT_FOUND_DURING_SCHEDULE_CONFIGURATION`: the tag's connector is not an active `pi_web_api` connector with `url` and `pi_web_api.auth` (`type`, `secret.name`, `secret.key`);
|
|
- `ORCHESTRATOR_PI_MIXED_CONNECTORS_DURING_SCHEDULE_CONFIGURATION`: the held pipelines above.
|
|
|
|
#### Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph "Parallel Data Loading"
|
|
A[1. aggregate_documents_in_mongodb]
|
|
B[2. find_documents_in_mongodb<br/>OPC Servers]
|
|
C[3. find_documents_in_mongodb<br/>Orchestrated Schedules]
|
|
D[4. load_opc_slots]
|
|
E[5. load_active_ingestors]
|
|
end
|
|
|
|
subgraph "Parallel Processing"
|
|
F[6. format_schedule_config]
|
|
G[7. process_schedules]
|
|
H[8. process_slots]
|
|
end
|
|
|
|
subgraph "Parallel Config Creation"
|
|
I[9. create_schedule_config]
|
|
J[10. create_slot_config]
|
|
K[11. normalize_schedules]
|
|
L[12. create_collection_with_ttl_index]
|
|
end
|
|
|
|
subgraph "Parallel Operations"
|
|
M[13. delete_slots]
|
|
N[14. update_slots]
|
|
O[15. delete_schedules]
|
|
P[16. create_schedules]
|
|
Q[17. update_schedules]
|
|
end
|
|
|
|
subgraph "Parallel Reports & Timestamps"
|
|
R[18. report_schedule_orchestration]
|
|
S[19. report_slot_orchestration]
|
|
T[20. update_pipelines_timestamps]
|
|
U[21. create_pipelines_timestamps]
|
|
V[22. delete_pipelines_timestamps]
|
|
end
|
|
|
|
A --> F
|
|
F --> I
|
|
I --> M
|
|
M --> R
|
|
|
|
A -.-> MongoDB1[(MongoDB)]
|
|
D -.-> Redis1[(Redis)]
|
|
K -.-> Temporal[(Temporal)]
|
|
R -.-> Reports[Reports]
|
|
```
|
|
|
|
Note: Green blocks (🟩) indicate parallel processing operations that run concurrently for improved performance.
|
|
|
|
|
|
#### 2. Alerts Workflow (`alerts.py`)
|
|
|
|
The **Alerts** workflow processes and sends real-time error notifications to configured user groups with intelligent filtering and duplicate prevention.
|
|
|
|
**Purpose**:
|
|
- **Error Alerting**: Immediate notification of ERROR-level events
|
|
- **TTL Management**: Prevents alert spam using configurable time-to-live settings
|
|
- **Group Filtering**: Sends alerts only to relevant user groups
|
|
- **Persistent Monitoring**: Tracks and escalates persistent issues
|
|
|
|
**Execution Flow**:
|
|
1. **Notification Loading**: Subworkflow loads ERROR-level notifications from MongoDB with timestamp filtering
|
|
2. **Configuration Loading**: Loads active receiver group configurations from MongoDB
|
|
3. **Alert Filtering**: Applies TTL-based filtering to identify core_alerts (new) and persistent_alerts (ongoing issues)
|
|
4. **Group Filtering**: Filters notifications by user group content policies and ignore lists
|
|
5. **Email Processing**: Subworkflow generates HTML emails and sends to receiver groups
|
|
6. **Cache Storage**: Stores sent notification cache in Redis with configurable TTL to prevent duplicates
|
|
7. **Audit Logging**: Logs delivery results to PostgreSQL for monitoring
|
|
|
|
**Input Parameters**:
|
|
```json
|
|
{
|
|
"schedule_name": "error_alerts",
|
|
"notification_ttl": 3600,
|
|
"sent_ttl": 7200
|
|
}
|
|
```
|
|
|
|
#### Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[1. load_notification_package🔃] --> B[2. filter_notification_alerts] --> C[3. process_notifications🔃]
|
|
|
|
A -.-> MongoDB[(MongoDB)]
|
|
A -.-> Redis[(Redis)]
|
|
B -.-> Filters[Report Filters]
|
|
C -.-> Email[(Email)]
|
|
C -.-> PostgreSQL[(PostgreSQL)]
|
|
|
|
style A fill:#000,color:#fff
|
|
style C fill:#000,color:#fff
|
|
```
|
|
|
|
#### 3. Reports Workflow (`reports.py`)
|
|
|
|
The **Reports** workflow generates and sends scheduled comprehensive reports to configured user groups.
|
|
|
|
**Purpose**:
|
|
- **Scheduled Reporting**: Regular summary reports of system activity
|
|
- **Comprehensive Coverage**: Includes all notification levels (not just errors)
|
|
- **Group Management**: Customizable reports per user group
|
|
- **Audit Trail**: Complete logging of report delivery
|
|
|
|
**Execution Flow**:
|
|
1. **Notification Loading**: Subworkflow loads all notifications from MongoDB (any level) with timestamp filtering
|
|
2. **Configuration Loading**: Loads active receiver group configurations from MongoDB
|
|
3. **Report Filtering**: Filters notifications by user group content policies (must include "reports") and ignore lists
|
|
4. **Email Processing**: Subworkflow generates HTML reports organized by notification level and model, then sends to receiver groups
|
|
5. **Audit Logging**: Logs delivery results to PostgreSQL for monitoring and tracking
|
|
|
|
**Input Parameters**:
|
|
```json
|
|
{
|
|
"schedule_name": "reports",
|
|
}
|
|
```
|
|
|
|
#### Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[1. load_notification_package🔃] --> B[2. filter_notification_reports] --> C[3. process_notifications🔃]
|
|
|
|
A -.-> MongoDB[(MongoDB)]
|
|
A -.-> Redis[(Redis)]
|
|
B -.-> Filters[Report Filters]
|
|
C -.-> Email[(Email)]
|
|
C -.-> PostgreSQL[(PostgreSQL)]
|
|
|
|
style A fill:#000,color:#fff
|
|
style C fill:#000,color:#fff
|
|
```
|
|
|
|
### Subworkflows
|
|
|
|
#### 1. Load Notification Package (`load_notification_package.py`)
|
|
|
|
**Purpose**: Centralized notification data loading and configuration management for both alerts and reports workflows.
|
|
|
|
**Key Features**:
|
|
- **Incremental Processing**: Uses Redis timestamps for efficient data loading
|
|
- **Configuration Management**: Loads active receiver group configurations
|
|
- **Data Validation**: Ensures complete data packages before processing
|
|
- **Timestamp Management**: Updates last processed timestamps
|
|
|
|
**Input Parameters**:
|
|
```json
|
|
{
|
|
"metadata": {"workflow_name": "alerts", "schedule_name": "error_alerts"},
|
|
"mail_type": "Alerts",
|
|
"base_data_filter": {"level": "ERROR"}
|
|
}
|
|
```
|
|
|
|
**Returns**:
|
|
- `last_timestamp` (str | None): Last processed timestamp
|
|
- `notification_package` (list[dict]): Retrieved notifications
|
|
- `sending_configs` (list[dict]): Active receiver group configurations
|
|
|
|
**Key Activities**:
|
|
- `get_last_data_timestamp`: Retrieves last processed timestamp from Redis for incremental processing
|
|
- `find_documents_in_mongodb`: Loads active receiver group configurations from MongoDB
|
|
- `load_latest_data`: Loads notifications with timestamp filtering from notification_queue collection
|
|
- `put_last_data_timestamp`: Updates last processed timestamp in Redis with 5-hour TTL
|
|
|
|
**Architecture**:
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph "Parallel Loading"
|
|
A[1. get_last_data_timestamp]
|
|
B[2. find_documents_in_mongodb<br/>Receiver Groups]
|
|
end
|
|
|
|
C[3. load_latest_data] --> D[4. put_last_data_timestamp]
|
|
|
|
A --> C
|
|
B --> D
|
|
|
|
A -.-> Redis1[(Redis)]
|
|
B -.-> MongoDB1[(MongoDB)]
|
|
C -.-> MongoDB2[(MongoDB)]
|
|
D -.-> Redis2[(Redis)]
|
|
```
|
|
|
|
#### 2. Process Notifications (`process_notifications.py`)
|
|
|
|
**Purpose**: Handles email generation, delivery, and audit logging for notification workflows.
|
|
|
|
**Key Features**:
|
|
- **HTML Generation**: Creates formatted email content for each receiver group using Jinja2 templates
|
|
- **Email Delivery**: Sends emails with attachment support and error handling
|
|
- **Audit Logging**: Records delivery status and metrics in PostgreSQL
|
|
- **Error Recovery**: Handles SMTP failures with detailed error reporting
|
|
- **Template Support**: Uses customizable HTML templates for different email types
|
|
|
|
**Input Parameters**:
|
|
```json
|
|
{
|
|
"metadata": {"workflow_name": "alerts", "schedule_name": "error_alerts"},
|
|
"mail_type": "Alerts",
|
|
"schema": "sientia_data",
|
|
"table_name": "log_report",
|
|
"notification_package": {
|
|
"group_name": "admin_team",
|
|
"members": ["admin@example.com"],
|
|
"notifications": [...]
|
|
}
|
|
}
|
|
```
|
|
|
|
**Returns**:
|
|
- `log_report` (list[dict]): Detailed delivery status for each notification
|
|
|
|
**Key Activities**:
|
|
- `build_email_html`: Generates HTML content using Jinja2 templates organized by notification level and model
|
|
- `send_email`: Delivers emails to receiver groups with attachment support and automatic SMTP reconnection
|
|
- `format_log_report`: Formats delivery results into DataFrame structure for database storage, aggregating by notification ID and trigger
|
|
- `export_data_to_postgres`: Stores audit logs in PostgreSQL with timestamp conversion
|
|
|
|
**Architecture**:
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[1. build_email_html] --> B[2. send_email] --> C[3. format_log_report] --> D[4. export_data_to_postgres]
|
|
|
|
A -.-> HTML[HTML Generator]
|
|
B -.-> SMTP[(SMTP)]
|
|
C -.-> Formatter[Log Formatter]
|
|
D -.-> PostgreSQL[(PostgreSQL)]
|
|
|
|
```
|
|
|
|
## 🔔 Notification Filtering System
|
|
|
|
The orchestrator includes an advanced notification filtering system that prevents alert spam and ensures relevant notifications reach the appropriate user groups.
|
|
|
|
### **Alert Filtering (`filter_notification_alerts`)**
|
|
- **Purpose**: Filters error-level notifications for immediate alerts
|
|
- **TTL Management**: Prevents duplicate alerts using configurable time-to-live settings
|
|
- **Persistent Detection**: Identifies ongoing issues that require escalation
|
|
- **Group-based Filtering**: Routes notifications to appropriate receiver groups
|
|
- **Ignore Lists**: Supports notification exclusion per group
|
|
|
|
### **Report Filtering (`filter_notification_reports`)**
|
|
- **Purpose**: Filters all notification levels for comprehensive scheduled reports
|
|
- **Comprehensive Coverage**: Includes ERROR, WARNING, INFO, and DEBUG levels
|
|
- **Group Customization**: Applies different content policies per receiver group with ignore list support
|
|
- **Scheduled Processing**: Designed for regular report generation without TTL-based duplicate prevention
|
|
- **Duplicate Prevention**: Prevents duplicate notifications within the same report using trigger and notification ID keys
|
|
|
|
### **Notification Caching (`store_notification_cache`)**
|
|
- **Purpose**: Manages Redis-based notification cache for TTL enforcement
|
|
- **TTL Support**: Configurable expiration times for different notification types
|
|
- **Duplicate Prevention**: Ensures notifications aren't sent repeatedly within TTL window
|
|
- **Key Management**: Uses structured keys for efficient cache lookups
|
|
|
|
### Key Components
|
|
|
|
#### **Worker (`orchestrator/worker/worker.py`)**
|
|
- **Purpose**: Main application orchestrator managing Temporal workers and task queues
|
|
- **Responsibilities**:
|
|
- Temporal client initialization and connection management with SDK metrics
|
|
- Worker lifecycle management and graceful shutdown via `sientia_do.temporal.worker.prepare_worker`
|
|
- Task queue configuration (orchestrator, alerts, reports) with dedicated workers on `<workflow>-<runtime>-queue`
|
|
- Prometheus metrics server initialization on HTTP_METRICS_PORT
|
|
- Temporal SDK metrics server initialization on HTTP_SDK_METRICS_PORT
|
|
- Notification handler setup and configuration
|
|
- **Key Features**:
|
|
- Multi-queue worker management with three dedicated workers (orchestrator, alerts, reports)
|
|
- `RUNTIME` env var (default `legacy`) passed to every `prepare_worker` call
|
|
- Sync blocking activities run on `prepare_worker`'s `activity_executor` thread pool; workflows stay `async def`
|
|
- Application health metrics (app_up gauge) for Kubernetes liveness/readiness probes
|
|
- Graceful shutdown with cleanup procedures for all connections and exit code propagation to Kubernetes
|
|
- Parallel worker execution using asyncio.gather
|
|
|
|
#### **Activities (`orchestrator/activities/`)**
|
|
- **Activities**: Main activity orchestrator combining all operations (TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres)
|
|
- **TemporalManager**: Temporal schedule CRUD operations across scouter and laborious namespaces with search attributes
|
|
- **SlotManager**: Redis-based OPC slot and cache management with notification filtering and TTL-based duplicate prevention
|
|
- **MongoDB**: Document operations, aggregations, timestamp management, and TTL index creation
|
|
- **Email**: SMTP operations with HTML generation, attachment support, and automatic reconnection handling
|
|
- **Formatters**: Configuration processing, slot distribution algorithms, notification filtering for reports, and schedule/slot orchestration reporting
|
|
- **Postgres**: PostgreSQL operations for audit logging and data export (via sientia-dataops-library)
|
|
|
|
#### **Utilities (`orchestrator/utils/`)**
|
|
- **Connectors Configuration**: Database and service configuration management from environment variables
|
|
- **Email Builder**: HTML email template generation and formatting using Jinja2 templates with support for alerts and reports
|
|
- **Orchestrator Functions**: Pipeline configuration transformation utilities supporting:
|
|
- `scouter`: OPC UA data collection workflows
|
|
- `pi_web_api_scouter`: PI Web API data collection workflows
|
|
- `predictions_batch`: ML prediction workflows with OPC write-back and multi-stage filtering
|
|
- `minimal_retrain`: Model retraining workflows with SQL queries
|
|
- `drift`: Data drift detection workflows
|
|
- `simple_metrics`: Model performance metrics computation workflows
|
|
- **Converters**: Data type conversion and validation utilities including frequency parsing for Temporal schedules
|
|
- **Templates**: HTML email templates for alerts and reports (email_template.html, general_template.html)
|
|
|
|
## 📋 Prerequisites
|
|
|
|
### System Requirements
|
|
- Python 3.11+
|
|
- Temporal server/cluster
|
|
- Redis server
|
|
- MongoDB server
|
|
- PostgreSQL database
|
|
- SMTP server access
|
|
|
|
**Note**: External dependencies must be available either through:
|
|
- Kubernetes cluster deployment
|
|
- Docker Compose setup
|
|
- Cloud-managed services
|
|
- Local installations
|
|
|
|
## 🚀 Installation
|
|
|
|
### Local Development Setup
|
|
|
|
1. **Clone the repository**
|
|
```bash
|
|
git clone <repository-url>
|
|
cd sientia-dataops-orchestrator_temporal
|
|
```
|
|
|
|
2. **Create virtual environment**
|
|
```bash
|
|
python3.11 -m venv venv
|
|
source ./venv/bin/activate
|
|
```
|
|
|
|
3. **Install dependencies**
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
4. **Configure environment variables**
|
|
```bash
|
|
# Set required environment variables for services
|
|
export TEMPORAL_HOST=localhost:7233
|
|
export REDIS_HOST=localhost
|
|
export MONGODB_URL=localhost:27017
|
|
export POSTGRES_HOST=localhost
|
|
# ... additional configuration
|
|
```
|
|
|
|
## 📦 How to Run
|
|
|
|
### Running the Orchestrator Application
|
|
|
|
Use the provided script to run the application locally:
|
|
|
|
```bash
|
|
# Make script executable (first time only)
|
|
chmod +x run_local.sh
|
|
|
|
# Run the application
|
|
./run_local.sh
|
|
```
|
|
|
|
The script will:
|
|
- Activate the virtual environment
|
|
- Load environment variables from `.env`
|
|
- Start the orchestrator worker application
|
|
|
|
### Running Tests and Coverage
|
|
|
|
Use the provided script to run tests with coverage:
|
|
|
|
```bash
|
|
# Make script executable (first time only)
|
|
chmod +x run_coverage.sh
|
|
|
|
# Run tests with coverage
|
|
./run_coverage.sh
|
|
```
|
|
|
|
### Manual Application Execution
|
|
|
|
For manual execution without scripts:
|
|
|
|
```bash
|
|
# Activate virtual environment
|
|
source ./venv/bin/activate
|
|
|
|
# Start the orchestrator worker
|
|
python -m orchestrator.worker.worker
|
|
```
|
|
|
|
## ⚙️ Configuration
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Description | Default | Required |
|
|
|----------|-------------|---------|----------|
|
|
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
|
|
| `TEMPORAL_NAMESPACE` | Default Temporal namespace | `default` | No |
|
|
| `TEMPORAL_SCOUTER_NAMESPACE` | Scouter workflow namespace | `scouter` | No |
|
|
| `TEMPORAL_LABORIOUS_NAMESPACE` | Laborious workflow namespace | `laborious` | No |
|
|
| `RUNTIME` | Runtime slice for orchestrator worker task queues (`orchestrator-<runtime>-queue`, etc.) | `legacy` | No |
|
|
| `ACTIVITY_EXECUTOR_MAX_WORKERS` | Thread pool size for sync activities (all workers) | `200` | No |
|
|
| `ORCHESTRATOR_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Orchestrator worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No |
|
|
| `ALERTS_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Alerts worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No |
|
|
| `REPORTS_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Reports worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No |
|
|
| `REDIS_HOST` | Redis server hostname | `localhost` | Yes |
|
|
| `REDIS_PORT` | Redis server port | `6379` | Yes |
|
|
| `REDIS_USERNAME` | Redis username | `default` | Yes |
|
|
| `REDIS_PASSWORD` | Redis password | - | Yes |
|
|
| `MONGODB_URL` | MongoDB server URL | `localhost:27017` | Yes |
|
|
| `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
|
|
| `MONGODB_PASSWORD` | MongoDB password | - | Yes |
|
|
| `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes |
|
|
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
|
|
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
|
|
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
|
|
| `POSTGRES_PASSWORD` | PostgreSQL password | - | Yes |
|
|
| `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
|
|
| `EMAIL_SENDER` | Sender email address | - | Yes |
|
|
| `EMAIL_SENDER_PASSWORD` | SMTP password | - | Yes |
|
|
| `EMAIL_SMTP_SERVER` | SMTP server | `smtp.gmail.com` | No |
|
|
| `EMAIL_SMTP_PORT` | SMTP port | `587` | No |
|
|
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
|
|
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
|
|
| `POSTGRES_MIN_CONNECTIONS` | PostgreSQL minimum connections | `10` | No |
|
|
| `POSTGRES_MAX_CONNECTIONS` | PostgreSQL maximum connections | `40` | No |
|
|
| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index expiration in hours | `1` | No |
|
|
| `PROJECT_NAME` | Project name for metrics and logging | `sientia-orchestrator` | No |
|
|
| `LOG_LEVEL` | Application logging level | `INFO` | No |
|
|
| `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No |
|
|
|
|
### Workflow Configuration
|
|
|
|
Temporal input configuration sample:
|
|
|
|
#### Orchestrator Workflow
|
|
|
|
```json
|
|
{
|
|
"schedule_name": "orchestrator-test",
|
|
"pipelines_query": {
|
|
"collection": "pipelines",
|
|
"aggregation": [
|
|
{
|
|
"$lookup": {
|
|
"from": "models",
|
|
"localField": "model_id",
|
|
"foreignField": "id",
|
|
"as": "model_docs"
|
|
}
|
|
},
|
|
{
|
|
"$match": {
|
|
"active": True
|
|
}
|
|
},
|
|
{
|
|
"$addFields": {
|
|
"models": {
|
|
"$arrayElemAt": [
|
|
"$model_docs",
|
|
0
|
|
]
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"$match": {
|
|
"models.active": True
|
|
}
|
|
},
|
|
{
|
|
"$project": {
|
|
"model_docs": 0
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"opc_servers_query": {
|
|
"collection": "opc-servers",
|
|
"filters": {
|
|
|
|
}
|
|
},
|
|
"connectors_query": {
|
|
"collection": "connectors",
|
|
"aggregation": [
|
|
{"$match": {"type": "opc_ua", "status": "Active"}},
|
|
{"$project": {"_id": 0, "connector_id": 1, "name": 1, "url": 1, "server_uri": 1,
|
|
"use_security": 1, "subscription_period_ms": 1}}
|
|
]
|
|
},
|
|
"connector_tags_query": {
|
|
"collection": "connector_tags",
|
|
"aggregation": [
|
|
{"$match": {"category": "Read", "status": "Active"}},
|
|
{"$project": {"_id": 0, "tag_id": 1, "connector_id": 1, "address": 1, "tag_name": 1}}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
`connectors_query` and `connector_tags_query` are optional: the values above are the defaults in the code (see "OPC slots" above).
|
|
|
|
#### Simple metrics: parallel windows & adaptive frequency
|
|
|
|
One `pipelines` doc per window, all with the same `model_id`. Name them `<model>-simple_metrics-<interval_minutes>m`; each doc becomes its own Temporal schedule. Thresholds are shared through `model.model_config`. The adaptive state is per doc.
|
|
|
|
```json
|
|
{
|
|
"schedule_name": "model-x-simple_metrics-60m",
|
|
"workflow_type": "simple_metrics",
|
|
"model_id": "model-x",
|
|
"active": true,
|
|
"interval_minutes": 60,
|
|
"frequency": "1h",
|
|
"frequency_alert": "5m"
|
|
}
|
|
```
|
|
|
|
| Field | Meaning | Default |
|
|
|---|---|---|
|
|
| `frequency_alert` | Frequency while escalated, same format as `frequency`. Absent = adaptive mode off. Below `1m` (or unparseable), the orchestrator sends a WARNING each time it creates or updates the schedule. A value below `1m` is kept; an unparseable value is ignored (the schedule keeps `frequency`) | — |
|
|
| `adaptive_state` | `{escalated, healthy_streak}`, written **only** by laborious (with `updated_at` as a BSON datetime when `escalated` flips) | `{false, 0}` |
|
|
| `model.model_config.simple_metrics_thresholds` | `{<metric>_max, <metric>_min}` shared by all windows | — |
|
|
| `model.model_config.simple_metrics_min_samples` | Windows with fewer rows neither alert nor escalate, and count toward recovery | — |
|
|
|
|
Escalation, recovery and the per-model alert are implemented in laborious (SIENTIAPDE-2189, SIENTIAPDE-2191); the orchestrator only picks `frequency_alert` while `adaptive_state.escalated` is true.
|
|
|
|
Rules:
|
|
- **Escalation:** laborious escalates on the first breach.
|
|
- **Recovery:** after `min(12, ceil(interval_minutes / frequency_alert))` consecutive runs without one. Empty windows count as runs without a breach.
|
|
- **Alerts** are per model (trigger `<model_id>-simple_metrics`), so several windows seeing the same breach produce one alert.
|
|
- **Delay:** the schedule switches on the orchestrator tick after laborious flips the state, at most one tick later.
|
|
|
|
#### Alerts Workflow
|
|
|
|
```json
|
|
{
|
|
"schedule_name": "alerts",
|
|
"notification_ttl": 5*60,
|
|
"sent_ttl": 10*60
|
|
}
|
|
```
|
|
|
|
#### Reports Workflow
|
|
|
|
```json
|
|
{
|
|
"schedule_name": "reports",
|
|
}
|
|
```
|
|
|
|
## 📊 Monitoring and Metrics
|
|
|
|
The Orchestrator system exposes comprehensive Prometheus metrics:
|
|
|
|
### Application Metrics
|
|
- `app_up`: Application health status gauge (1=healthy, 0=unhealthy) labeled by pod_id
|
|
- `email_sent_count`: Email delivery counter labeled by pod_id, model_name, pipeline_name, and email_group
|
|
|
|
### Workflow Metrics
|
|
- Schedule creation, update, and deletion success rates (via notification reports)
|
|
- Notification processing times and error rates (via PostgreSQL audit logs)
|
|
- Resource allocation and slot management metrics (via notification reports)
|
|
- Temporal SDK metrics exposed on HTTP_SDK_METRICS_PORT (default: 9091)
|
|
|
|
### Database Metrics
|
|
- MongoDB query performance and connection health (via sientia-dataops-library)
|
|
- Redis operation counts and response times (via sientia-dataops-library)
|
|
- PostgreSQL export operations and audit log metrics (via sientia-dataops-library)
|
|
|
|
## 🧪 Testing
|
|
|
|
### Test Structure
|
|
```
|
|
tests/
|
|
├── orchestrator/ # Orchestrator workflow tests
|
|
│ ├── test_activities.py
|
|
│ ├── test_email.py
|
|
│ ├── test_formatters.py
|
|
│ ├── test_mongo_db.py
|
|
│ ├── test_slot_manager.py
|
|
│ ├── test_temporal_manager.py
|
|
│ └── test_workflows.py
|
|
├── activities/ # Activity implementation tests
|
|
├── utils/ # Utility function tests
|
|
└── integration/ # End-to-end workflow tests
|
|
```
|
|
|
|
### Test Coverage
|
|
The project maintains comprehensive test coverage including:
|
|
- **Activity Tests**: Unit tests for all activity classes
|
|
- **Workflow Tests**: Integration tests for workflow orchestration
|
|
- **Utility Tests**: Tests for configuration builders and converters
|
|
- **Database Tests**: Tests for MongoDB, Redis, and PostgreSQL operations
|
|
- **Email Tests**: Tests for email generation and delivery
|
|
- **Notification Tests**: Tests for filtering and caching logic
|
|
|
|
### Test Execution
|
|
```bash
|
|
# Install test dependencies
|
|
pip install -r requirements-dev.txt
|
|
|
|
# Run unit tests with coverage (default testpaths=tests; E2E excluded)
|
|
pytest --cov=orchestrator --cov-report=html
|
|
|
|
# Run specific test modules
|
|
pytest tests/orchestrator/activities/test_mongo_db.py
|
|
pytest tests/orchestrator/workflows/test_orchestrator.py
|
|
```
|
|
|
|
### End-to-end tests
|
|
|
|
E2E tests live in `e2e/` and require **Docker** (testcontainers). They are **not** collected by default `pytest` at the repo root.
|
|
|
|
```bash
|
|
source ./venv/bin/activate
|
|
pip install -r requirements-dev.txt
|
|
pytest e2e/ --override-ini testpaths=e2e -m e2e -v
|
|
```
|
|
|
|
Coverage is kept separate from unit tests: set `COVERAGE_FILE=.coverage.e2e` when measuring E2E coverage (see `e2e/README.md`).
|
|
|
|
## 🛡️ Code Quality & Validation
|
|
|
|
### Overview
|
|
|
|
Since Python is not compiled, this project ships a validation workflow to catch issues early. Use the `validate.sh` script to run formatting, linting, type checks, security analysis, and tests in one command.
|
|
|
|
### Validation Tools
|
|
|
|
- **Ruff**: formatting and linting (fast, replaces Black/Flake8)
|
|
- **mypy**: static typing checks
|
|
- **Bandit**: security static analysis
|
|
- **pytest**: unit/integration tests with coverage
|
|
|
|
### Tools Installation
|
|
|
|
```bash
|
|
pip install -r requirements-dev.txt
|
|
```
|
|
|
|
### Complete Validation (recommended)
|
|
|
|
```bash
|
|
./validate.sh
|
|
```
|
|
|
|
What `validate.sh` does:
|
|
1. Checks formatting with Ruff
|
|
2. Lints code with Ruff
|
|
3. Runs mypy type checking
|
|
4. Runs Bandit security analysis
|
|
5. Executes pytest with coverage (generates HTML report)
|
|
|
|
Exit codes are propagated so CI can fail fast when quality gates are not met.
|
|
|
|
### Individual Commands
|
|
|
|
```bash
|
|
# 1) Format check
|
|
ruff format --check orchestrator/ tests/
|
|
|
|
# 2) Lint
|
|
ruff check orchestrator/ tests/
|
|
|
|
# 3) Type check
|
|
mypy orchestrator/
|
|
|
|
# 4) Security
|
|
bandit -r orchestrator/ -ll
|
|
|
|
# 5) Tests with coverage
|
|
pytest tests/ --cov=orchestrator --cov-report=html
|
|
```
|
|
|
|
### Automatic Fixes
|
|
|
|
```bash
|
|
# Apply formatting
|
|
ruff format orchestrator/ tests/
|
|
|
|
# Autofix common lint issues
|
|
ruff check --fix orchestrator/ tests/
|
|
```
|
|
|
|
### Configuration
|
|
|
|
Tooling is configured in `pyproject.toml` (lint rules, formatting, typing). Adjust thresholds and rules there as needed.
|
|
|
|
## 🔧 Development
|
|
|
|
### Project Structure
|
|
```
|
|
orchestrator/
|
|
├── activities/ # Temporal activity implementations
|
|
│ ├── activities.py # Main activities orchestrator
|
|
│ ├── temporal_manager.py # Temporal schedule operations
|
|
│ ├── slot_manager.py # Redis slot management and notification filtering
|
|
│ ├── mongo_db.py # MongoDB operations
|
|
│ ├── email.py # Email service operations
|
|
│ ├── formatters.py # Configuration formatting and report filtering
|
|
├── workflows/ # Temporal workflow definitions
|
|
│ ├── orchestrator.py # Main orchestration workflow
|
|
│ ├── alerts.py # Error alert workflow
|
|
│ ├── reports.py # Scheduled report workflow
|
|
│ └── subworkflows/ # Sub-workflow implementations
|
|
│ ├── load_notification_package.py # Notification data loading
|
|
│ └── process_notifications.py # Email processing and delivery
|
|
├── worker/ # Worker implementation
|
|
│ └── worker.py # Main worker orchestrator
|
|
├── utils/ # Utility functions
|
|
│ ├── connectors_config.py # Database configuration
|
|
│ ├── email_builder.py # Email template generation
|
|
│ ├── orchestrator_functions.py # Pipeline utilities
|
|
│ ├── converters.py # Data type conversion utilities
|
|
│ └── templates/ # Email HTML templates
|
|
│ ├── email_template.html # Report email template
|
|
│ └── general_template.html # General email template
|
|
└── metrics.py # Prometheus metrics definitions
|
|
```
|
|
|
|
### Adding New Features
|
|
|
|
1. **Follow Temporal patterns** for new workflows and activities
|
|
2. **Add comprehensive docstrings** for all public methods
|
|
3. **Include Prometheus metrics** for monitoring
|
|
4. **Add unit tests** for new functionality
|
|
5. **Update this README** with new features and configuration
|
|
|
|
## 🐛 Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Temporal Connection Failures**
|
|
- Verify Temporal server is running and accessible
|
|
- Check namespace configuration and permissions
|
|
- Review server logs for connection issues
|
|
|
|
2. **Database Connection Issues**
|
|
- Verify all database services are running
|
|
- Check connection credentials and network access
|
|
- Ensure proper connection pool configuration
|
|
|
|
3. **Email Delivery Failures**
|
|
- Verify SMTP server configuration and credentials
|
|
- Check email sender permissions and authentication
|
|
- Review email delivery logs for specific errors
|
|
|
|
4. **Workflow Execution Failures**
|
|
- Review activity error logs and notifications
|
|
- Check MongoDB collection configurations
|
|
- Verify input data format and required fields
|
|
|
|
5. **E2E / testcontainers leftovers**
|
|
- If a run is interrupted, containers may keep running. List and remove them:
|
|
```bash
|
|
docker ps -a --filter label=org.testcontainers=true
|
|
docker rm -f $(docker ps -aq --filter label=org.testcontainers=true)
|
|
```
|
|
|
|
### Debug Mode
|
|
|
|
Enable debug logging by setting the log level:
|
|
```bash
|
|
export LOG_LEVEL=DEBUG
|
|
```
|
|
|
|
## ⚡ Performance Tuning
|
|
|
|
### Key Parameters
|
|
|
|
- **Worker Concurrency**: Configure worker task limits in Temporal client
|
|
- **Connection Pools**: Optimize database connection pool sizes
|
|
- **Redis TTL**: Adjust cache TTL settings based on requirements
|
|
- **Batch Sizes**: Configure notification processing batch sizes
|
|
|
|
### Scaling Considerations
|
|
|
|
- **Horizontal Scaling**: Deploy multiple worker instances
|
|
- **Task Queue Distribution**: Use dedicated queues for different workflows
|
|
- **Database Performance**: Optimize indexes and connection pooling
|
|
- **Memory Management**: Monitor and configure appropriate resource limits
|
|
|
|
## 🤝 Contributing
|
|
|
|
1. Fork the repository
|
|
2. Create a feature branch
|
|
3. Make your changes with comprehensive testing
|
|
4. Update documentation and docstrings
|
|
5. Submit a pull request
|
|
|
|
### Code Quality Standards
|
|
|
|
- Follow PEP 8 style guidelines
|
|
- Include comprehensive docstrings for all public methods
|
|
- Maintain test coverage above 80%
|
|
- Use type hints where appropriate
|
|
- Follow Temporal.io best practices
|
|
|
|
## 📄 License
|
|
|
|
This project is licensed under the terms specified in the LICENSE file.
|
|
|
|
## 🆘 Support
|
|
|
|
For support and questions:
|
|
- Check the troubleshooting section above
|
|
- Review the metrics and logs for error patterns
|
|
- Open an issue in the project repository
|
|
- Contact the development team
|
|
|
|
---
|
|
|
|
### Runtime queues and migration
|
|
|
|
- **Queue naming**: Downstream pipeline schedules target `<workflow_type>-<runtime>-queue` (for example `scouter-legacy-queue`). The orchestrator's own workers register on `orchestrator-<runtime>-queue`, `alerts-<runtime>-queue`, and `reports-<runtime>-queue`.
|
|
- **Pipeline `runtime`**: Each pipeline JSON may include `runtime` (default `legacy` via `common_config`). Schedules without `runtime` use `legacy`.
|
|
- **Schedule updates**: Changing a pipeline's `runtime` does not update an existing schedule's `task_queue` in place; the next orchestrator tick deletes and recreates the schedule on the new queue.
|
|
- **Rollout order**: Deploy scouter/laborious worker fleets bound to the new `<workflow_type>-<runtime>-queue` family before redeploying the orchestrator. On the first tick after upgrade, orphan schedules on old queue names are normalized away and schedules are recreated on the new queues (no in-place rename).
|
|
|
|
**Note**: The SIENTIA DataOps Orchestrator is designed for production use in enterprise data environments. Ensure proper security configuration and network isolation for production deployments.
|