# SIENTIA DataOps Orchestrator Temporal A high-performance, scalable workflow orchestration system built on Temporal.io for automated pipeline management, notification delivery, and resource coordination. The Orchestrator provides enterprise-grade workflow automation, real-time alerting, and comprehensive monitoring capabilities for the SIENTIA platform. ## πŸ“‘ Table of Contents - [Features](#features) - [Core Functionality](#core-functionality) - [Advanced Capabilities](#advanced-capabilities) - [Architecture](#architecture) - [Architecture Principles](#architecture-principles) - [Task Queue Isolation](#2-task-queue-isolation) - [Workflows](#-workflows) - [Orchestrator Workflow](#1-orchestrator-workflow-orchestratorpy) - [Alerts Workflow](#2-alerts-workflow-alertspy) - [Reports Workflow](#3-reports-workflow-reportspy) - [Subworkflows](#subworkflows) - [Load Notification Package](#1-load-notification-package-load_notification_packagepy) - [Process Notifications](#2-process-notifications-process_notificationspy) - [Notification Filtering System](#-notification-filtering-system) - [Prerequisites](#-prerequisites) - [Installation](#-installation) - [How to Run](#-how-to-run) - [Configuration](#-configuration) - [Monitoring and Metrics](#-monitoring-and-metrics) - [Testing](#-testing) - [Code Quality & Validation](#-code-quality--validation) - [Development](#-development) - [Troubleshooting](#-troubleshooting) - [Performance Tuning](#-performance-tuning) - [Contributing](#-contributing) - [License](#-license) - [Support](#-support) ## Features ### Core Functionality - **Pipeline Orchestration**: Automated deployment and management of data processing pipelines - **Real-time Notifications**: Intelligent alert filtering and delivery with TTL management - **Resource Management**: Dynamic OPC server slot allocation and active ingestor monitoring - **Schedule Management**: Temporal-based workflow scheduling with automatic retry policies - **Multi-namespace Support**: Separate workflow queues for scouter and laborious operations ### Advanced Capabilities - **Incremental Processing**: Timestamp-based data loading to avoid reprocessing - **Intelligent Notification Filtering**: Advanced filtering system with: - User group-based notification filtering with custom policies - TTL-based duplicate prevention for alerts - Ignore lists for specific notifications - Persistent alert detection for ongoing issues - **Auto-scaling Workers**: Multiple worker instances with task queue isolation - **Comprehensive Logging**: Structured logging with PostgreSQL audit trails - **Prometheus Metrics**: Real-time monitoring and alerting integration - **Template-based Email Generation**: Jinja2-powered HTML email templates ## Architecture The SIENTIA DataOps Orchestrator uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in data pipeline environments. ### Architecture Principles #### 1. **Separation of Concerns** - **Worker Layer**: Manages Temporal workers, task queues, Prometheus metrics, and application lifecycle - **Workflow Layer**: Orchestrates business logic and process coordination with subworkflows for modularity - **Activity Layer**: Implements specific operations and external system interactions with error handling - **Data Layer**: Handles data persistence (MongoDB, PostgreSQL), caching (Redis), and external service connections #### 2. **Task Queue Isolation** - **Orchestrator Queue**: Pipeline and resource management workflows (orchestrator workflow) - **Alerts Queue**: Real-time error notification workflows (alerts workflow, load_notification_package, process_notifications subworkflows) - **Reports Queue**: Scheduled reporting and summary workflows (reports workflow, load_notification_package, process_notifications subworkflows) #### 3. **Fault Tolerance & Resilience** - **Automatic Retry Policies**: Configurable retry strategies for transient failures - **Graceful Degradation**: System continues operating with reduced functionality - **Comprehensive Error Handling**: Detailed error reporting and notification integration - **Connection Management**: Automatic reconnection for SMTP and database services #### 4. **Scalability & Performance** - **Horizontal Scaling**: Multiple worker instances for load distribution - **Connection Pooling**: Optimized database and Redis connections - **Asynchronous Processing**: Non-blocking operations for improved throughput - **Resource Optimization**: Intelligent slot allocation and ingestor management ## πŸ”„ Workflows ### Main Workflows #### 1. Orchestrator Workflow (`orchestrator.py`) The **Orchestrator** workflow is the main coordination workflow that manages pipeline deployment and resource allocation across the SIENTIA platform. **Purpose**: - **Pipeline Management**: Coordinates deployment of scouter and laborious pipelines - **Resource Allocation**: Manages OPC server slots and active ingestor distribution - **Schedule Synchronization**: Ensures Temporal schedules match MongoDB configurations - **Infrastructure Management**: Creates, updates, and deletes workflow schedules **Execution Flow**: 1. **Parallel Data Loading**: Concurrently loads pipelines, OPC servers, orchestrated schedules, OPC slots, and active ingestors 2. **Parallel Processing**: Formats orchestrated schedules and processes new schedules and slots 3. **Parallel Config Creation**: Creates schedule and slot action configurations, normalizes schedules, and creates collections with TTL indexes 4. **Parallel Operations**: Executes slot deletions, slot updates, schedule deletions, schedule creations, and schedule updates concurrently 5. **Parallel Reports & Timestamps**: Generates orchestration reports and updates MongoDB timestamps for created/updated/deleted pipelines **Input Parameters**: ```json { "schedule_name": "hourly_orchestration", "pipelines_query": { "collection": "pipelines", "aggregation": [ {"$match": {"active": true}}, {"$sort": {"updated_at": -1}} ] }, "opc_servers_query": { "collection": "opc_servers", "filters": {"active": true} } } ``` #### Architecture ```mermaid flowchart LR subgraph "Parallel Data Loading" A[1. aggregate_documents_in_mongodb] B[2. find_documents_in_mongodb
OPC Servers] C[3. find_documents_in_mongodb
Orchestrated Schedules] D[4. load_opc_slots] E[5. load_active_ingestors] end subgraph "Parallel Processing" F[6. format_schedule_config] G[7. process_schedules] H[8. process_slots] end subgraph "Parallel Config Creation" I[9. create_schedule_config] J[10. create_slot_config] K[11. normalize_schedules] L[12. create_collection_with_ttl_index] end subgraph "Parallel Operations" M[13. delete_slots] N[14. update_slots] O[15. delete_schedules] P[16. create_schedules] Q[17. update_schedules] end subgraph "Parallel Reports & Timestamps" R[18. report_schedule_orchestration] S[19. report_slot_orchestration] T[20. update_pipelines_timestamps] U[21. create_pipelines_timestamps] V[22. delete_pipelines_timestamps] end A --> F F --> I I --> M M --> R A -.-> MongoDB1[(MongoDB)] D -.-> Redis1[(Redis)] K -.-> Temporal[(Temporal)] R -.-> Reports[Reports] ``` Note: Green blocks (🟩) indicate parallel processing operations that run concurrently for improved performance. #### 2. Alerts Workflow (`alerts.py`) The **Alerts** workflow processes and sends real-time error notifications to configured user groups with intelligent filtering and duplicate prevention. **Purpose**: - **Error Alerting**: Immediate notification of ERROR-level events - **TTL Management**: Prevents alert spam using configurable time-to-live settings - **Group Filtering**: Sends alerts only to relevant user groups - **Persistent Monitoring**: Tracks and escalates persistent issues **Execution Flow**: 1. **Notification Loading**: Subworkflow loads ERROR-level notifications from MongoDB with timestamp filtering 2. **Configuration Loading**: Loads active receiver group configurations from MongoDB 3. **Alert Filtering**: Applies TTL-based filtering to identify core_alerts (new) and persistent_alerts (ongoing issues) 4. **Group Filtering**: Filters notifications by user group content policies and ignore lists 5. **Email Processing**: Subworkflow generates HTML emails and sends to receiver groups 6. **Cache Storage**: Stores sent notification cache in Redis with configurable TTL to prevent duplicates 7. **Audit Logging**: Logs delivery results to PostgreSQL for monitoring **Input Parameters**: ```json { "schedule_name": "error_alerts", "notification_ttl": 3600, "sent_ttl": 7200 } ``` #### Architecture ```mermaid flowchart LR A[1. load_notification_packageπŸ”ƒ] --> B[2. filter_notification_alerts] --> C[3. process_notificationsπŸ”ƒ] A -.-> MongoDB[(MongoDB)] A -.-> Redis[(Redis)] B -.-> Filters[Report Filters] C -.-> Email[(Email)] C -.-> PostgreSQL[(PostgreSQL)] style A fill:#000,color:#fff style C fill:#000,color:#fff ``` #### 3. Reports Workflow (`reports.py`) The **Reports** workflow generates and sends scheduled comprehensive reports to configured user groups. **Purpose**: - **Scheduled Reporting**: Regular summary reports of system activity - **Comprehensive Coverage**: Includes all notification levels (not just errors) - **Group Management**: Customizable reports per user group - **Audit Trail**: Complete logging of report delivery **Execution Flow**: 1. **Notification Loading**: Subworkflow loads all notifications from MongoDB (any level) with timestamp filtering 2. **Configuration Loading**: Loads active receiver group configurations from MongoDB 3. **Report Filtering**: Filters notifications by user group content policies (must include "reports") and ignore lists 4. **Email Processing**: Subworkflow generates HTML reports organized by notification level and model, then sends to receiver groups 5. **Audit Logging**: Logs delivery results to PostgreSQL for monitoring and tracking **Input Parameters**: ```json { "schedule_name": "reports", } ``` #### Architecture ```mermaid flowchart LR A[1. load_notification_packageπŸ”ƒ] --> B[2. filter_notification_reports] --> C[3. process_notificationsπŸ”ƒ] A -.-> MongoDB[(MongoDB)] A -.-> Redis[(Redis)] B -.-> Filters[Report Filters] C -.-> Email[(Email)] C -.-> PostgreSQL[(PostgreSQL)] style A fill:#000,color:#fff style C fill:#000,color:#fff ``` ### Subworkflows #### 1. Load Notification Package (`load_notification_package.py`) **Purpose**: Centralized notification data loading and configuration management for both alerts and reports workflows. **Key Features**: - **Incremental Processing**: Uses Redis timestamps for efficient data loading - **Configuration Management**: Loads active receiver group configurations - **Data Validation**: Ensures complete data packages before processing - **Timestamp Management**: Updates last processed timestamps **Input Parameters**: ```json { "metadata": {"workflow_name": "alerts", "schedule_name": "error_alerts"}, "mail_type": "Alerts", "base_data_filter": {"level": "ERROR"} } ``` **Returns**: - `last_timestamp` (str | None): Last processed timestamp - `notification_package` (list[dict]): Retrieved notifications - `sending_configs` (list[dict]): Active receiver group configurations **Key Activities**: - `get_last_data_timestamp`: Retrieves last processed timestamp from Redis for incremental processing - `find_documents_in_mongodb`: Loads active receiver group configurations from MongoDB - `load_latest_data`: Loads notifications with timestamp filtering from notification_queue collection - `put_last_data_timestamp`: Updates last processed timestamp in Redis with 5-hour TTL **Architecture**: ```mermaid flowchart LR subgraph "Parallel Loading" A[1. get_last_data_timestamp] B[2. find_documents_in_mongodb
Receiver Groups] end C[3. load_latest_data] --> D[4. put_last_data_timestamp] A --> C B --> D A -.-> Redis1[(Redis)] B -.-> MongoDB1[(MongoDB)] C -.-> MongoDB2[(MongoDB)] D -.-> Redis2[(Redis)] ``` #### 2. Process Notifications (`process_notifications.py`) **Purpose**: Handles email generation, delivery, and audit logging for notification workflows. **Key Features**: - **HTML Generation**: Creates formatted email content for each receiver group using Jinja2 templates - **Email Delivery**: Sends emails with attachment support and error handling - **Audit Logging**: Records delivery status and metrics in PostgreSQL - **Error Recovery**: Handles SMTP failures with detailed error reporting - **Template Support**: Uses customizable HTML templates for different email types **Input Parameters**: ```json { "metadata": {"workflow_name": "alerts", "schedule_name": "error_alerts"}, "mail_type": "Alerts", "schema": "sientia_data", "table_name": "log_report", "notification_package": { "group_name": "admin_team", "members": ["admin@example.com"], "notifications": [...] } } ``` **Returns**: - `log_report` (list[dict]): Detailed delivery status for each notification **Key Activities**: - `build_email_html`: Generates HTML content using Jinja2 templates organized by notification level and model - `send_email`: Delivers emails to receiver groups with attachment support and automatic SMTP reconnection - `format_log_report`: Formats delivery results into DataFrame structure for database storage, aggregating by notification ID and trigger - `export_data_to_postgres`: Stores audit logs in PostgreSQL with timestamp conversion **Architecture**: ```mermaid flowchart LR A[1. build_email_html] --> B[2. send_email] --> C[3. format_log_report] --> D[4. export_data_to_postgres] A -.-> HTML[HTML Generator] B -.-> SMTP[(SMTP)] C -.-> Formatter[Log Formatter] D -.-> PostgreSQL[(PostgreSQL)] ``` ## πŸ”” Notification Filtering System The orchestrator includes an advanced notification filtering system that prevents alert spam and ensures relevant notifications reach the appropriate user groups. ### **Alert Filtering (`filter_notification_alerts`)** - **Purpose**: Filters error-level notifications for immediate alerts - **TTL Management**: Prevents duplicate alerts using configurable time-to-live settings - **Persistent Detection**: Identifies ongoing issues that require escalation - **Group-based Filtering**: Routes notifications to appropriate receiver groups - **Ignore Lists**: Supports notification exclusion per group ### **Report Filtering (`filter_notification_reports`)** - **Purpose**: Filters all notification levels for comprehensive scheduled reports - **Comprehensive Coverage**: Includes ERROR, WARNING, INFO, and DEBUG levels - **Group Customization**: Applies different content policies per receiver group with ignore list support - **Scheduled Processing**: Designed for regular report generation without TTL-based duplicate prevention - **Duplicate Prevention**: Prevents duplicate notifications within the same report using trigger and notification ID keys ### **Notification Caching (`store_notification_cache`)** - **Purpose**: Manages Redis-based notification cache for TTL enforcement - **TTL Support**: Configurable expiration times for different notification types - **Duplicate Prevention**: Ensures notifications aren't sent repeatedly within TTL window - **Key Management**: Uses structured keys for efficient cache lookups ### Key Components #### **Worker (`orchestrator/worker/worker.py`)** - **Purpose**: Main application orchestrator managing Temporal workers and task queues - **Responsibilities**: - Temporal client initialization and connection management with SDK metrics - Worker lifecycle management and graceful shutdown via `sientia_do.temporal.worker.prepare_worker` - Task queue configuration (orchestrator, alerts, reports) with dedicated workers on `--queue` - Prometheus metrics server initialization on HTTP_METRICS_PORT - Temporal SDK metrics server initialization on HTTP_SDK_METRICS_PORT - Notification handler setup and configuration - **Key Features**: - Multi-queue worker management with three dedicated workers (orchestrator, alerts, reports) - `RUNTIME` env var (default `legacy`) passed to every `prepare_worker` call - Sync blocking activities run on `prepare_worker`'s `activity_executor` thread pool; workflows stay `async def` - Application health metrics (app_up gauge) for Kubernetes liveness/readiness probes - Graceful shutdown with cleanup procedures for all connections and exit code propagation to Kubernetes - Parallel worker execution using asyncio.gather #### **Activities (`orchestrator/activities/`)** - **Activities**: Main activity orchestrator combining all operations (TemporalManager, SlotManager, Formatters, MongoDB, Email, Postgres) - **TemporalManager**: Temporal schedule CRUD operations across scouter and laborious namespaces with search attributes - **SlotManager**: Redis-based OPC slot and cache management with notification filtering and TTL-based duplicate prevention - **MongoDB**: Document operations, aggregations, timestamp management, and TTL index creation - **Email**: SMTP operations with HTML generation, attachment support, and automatic reconnection handling - **Formatters**: Configuration processing, slot distribution algorithms, notification filtering for reports, and schedule/slot orchestration reporting - **Postgres**: PostgreSQL operations for audit logging and data export (via sientia-dataops-library) #### **Utilities (`orchestrator/utils/`)** - **Connectors Configuration**: Database and service configuration management from environment variables - **Email Builder**: HTML email template generation and formatting using Jinja2 templates with support for alerts and reports - **Orchestrator Functions**: Pipeline configuration transformation utilities supporting: - `scouter`: OPC UA data collection workflows - `pi_web_api_scouter`: PI Web API data collection workflows - `predictions_batch`: ML prediction workflows with OPC write-back and multi-stage filtering - `minimal_retrain`: Model retraining workflows with SQL queries - `drift`: Data drift detection workflows - `simple_metrics`: Model performance metrics computation workflows - **Converters**: Data type conversion and validation utilities including frequency parsing for Temporal schedules - **Templates**: HTML email templates for alerts and reports (email_template.html, general_template.html) ## πŸ“‹ Prerequisites ### System Requirements - Python 3.11+ - Temporal server/cluster - Redis server - MongoDB server - PostgreSQL database - SMTP server access **Note**: External dependencies must be available either through: - Kubernetes cluster deployment - Docker Compose setup - Cloud-managed services - Local installations ## πŸš€ Installation ### Local Development Setup 1. **Clone the repository** ```bash git clone cd sientia-dataops-orchestrator_temporal ``` 2. **Create virtual environment** ```bash python3.11 -m venv venv source ./venv/bin/activate ``` 3. **Install dependencies** ```bash pip install -r requirements.txt ``` 4. **Configure environment variables** ```bash # Set required environment variables for services export TEMPORAL_HOST=localhost:7233 export REDIS_HOST=localhost export MONGODB_URL=localhost:27017 export POSTGRES_HOST=localhost # ... additional configuration ``` ## πŸ“¦ How to Run ### Running the Orchestrator Application Use the provided script to run the application locally: ```bash # Make script executable (first time only) chmod +x run_local.sh # Run the application ./run_local.sh ``` The script will: - Activate the virtual environment - Load environment variables from `.env` - Start the orchestrator worker application ### Running Tests and Coverage Use the provided script to run tests with coverage: ```bash # Make script executable (first time only) chmod +x run_coverage.sh # Run tests with coverage ./run_coverage.sh ``` ### Manual Application Execution For manual execution without scripts: ```bash # Activate virtual environment source ./venv/bin/activate # Start the orchestrator worker python -m orchestrator.worker.worker ``` ## βš™οΈ Configuration ### Environment Variables | Variable | Description | Default | Required | |----------|-------------|---------|----------| | `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes | | `TEMPORAL_NAMESPACE` | Default Temporal namespace | `default` | No | | `TEMPORAL_SCOUTER_NAMESPACE` | Scouter workflow namespace | `scouter` | No | | `TEMPORAL_LABORIOUS_NAMESPACE` | Laborious workflow namespace | `laborious` | No | | `RUNTIME` | Runtime slice for orchestrator worker task queues (`orchestrator--queue`, etc.) | `legacy` | No | | `ACTIVITY_EXECUTOR_MAX_WORKERS` | Thread pool size for sync activities (all workers) | `200` | No | | `ORCHESTRATOR_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Orchestrator worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No | | `ALERTS_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Alerts worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No | | `REPORTS_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Reports worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No | | `REDIS_HOST` | Redis server hostname | `localhost` | Yes | | `REDIS_PORT` | Redis server port | `6379` | Yes | | `REDIS_USERNAME` | Redis username | `default` | Yes | | `REDIS_PASSWORD` | Redis password | - | Yes | | `MONGODB_URL` | MongoDB server URL | `localhost:27017` | Yes | | `MONGODB_USERNAME` | MongoDB username | `root` | Yes | | `MONGODB_PASSWORD` | MongoDB password | - | Yes | | `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes | | `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes | | `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes | | `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes | | `POSTGRES_PASSWORD` | PostgreSQL password | - | Yes | | `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes | | `EMAIL_SENDER` | Sender email address | - | Yes | | `EMAIL_SENDER_PASSWORD` | SMTP password | - | Yes | | `EMAIL_SMTP_SERVER` | SMTP server | `smtp.gmail.com` | No | | `EMAIL_SMTP_PORT` | SMTP port | `587` | No | | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No | | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No | | `POSTGRES_MIN_CONNECTIONS` | PostgreSQL minimum connections | `10` | No | | `POSTGRES_MAX_CONNECTIONS` | PostgreSQL maximum connections | `40` | No | | `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index expiration in hours | `1` | No | | `PROJECT_NAME` | Project name for metrics and logging | `sientia-orchestrator` | No | | `LOG_LEVEL` | Application logging level | `INFO` | No | | `KAFKA_BOOTSTRAP_SERVERS` | Kafka bootstrap servers | - | No | ### Workflow Configuration Temporal input configuration sample: #### Orchestrator Workflow ```json { "schedule_name": "orchestrator-test", "pipelines_query": { "collection": "pipelines", "aggregation": [ { "$lookup": { "from": "models", "localField": "model_id", "foreignField": "id", "as": "model_docs" } }, { "$match": { "active": True } }, { "$addFields": { "models": { "$arrayElemAt": [ "$model_docs", 0 ] } } }, { "$match": { "models.active": True } }, { "$project": { "model_docs": 0 } } ] }, "opc_servers_query": { "collection": "opc-servers", "filters": { } } } ``` #### Alerts Workflow ```json { "schedule_name": "alerts", "notification_ttl": 5*60, "sent_ttl": 10*60 } ``` #### Reports Workflow ```json { "schedule_name": "reports", } ``` ## πŸ“Š Monitoring and Metrics The Orchestrator system exposes comprehensive Prometheus metrics: ### Application Metrics - `app_up`: Application health status gauge (1=healthy, 0=unhealthy) labeled by pod_id - `email_sent_count`: Email delivery counter labeled by pod_id, model_name, pipeline_name, and email_group ### Workflow Metrics - Schedule creation, update, and deletion success rates (via notification reports) - Notification processing times and error rates (via PostgreSQL audit logs) - Resource allocation and slot management metrics (via notification reports) - Temporal SDK metrics exposed on HTTP_SDK_METRICS_PORT (default: 9091) ### Database Metrics - MongoDB query performance and connection health (via sientia-dataops-library) - Redis operation counts and response times (via sientia-dataops-library) - PostgreSQL export operations and audit log metrics (via sientia-dataops-library) ## πŸ§ͺ Testing ### Test Structure ``` tests/ β”œβ”€β”€ orchestrator/ # Orchestrator workflow tests β”‚ β”œβ”€β”€ test_activities.py β”‚ β”œβ”€β”€ test_email.py β”‚ β”œβ”€β”€ test_formatters.py β”‚ β”œβ”€β”€ test_mongo_db.py β”‚ β”œβ”€β”€ test_slot_manager.py β”‚ β”œβ”€β”€ test_temporal_manager.py β”‚ └── test_workflows.py β”œβ”€β”€ activities/ # Activity implementation tests β”œβ”€β”€ utils/ # Utility function tests └── integration/ # End-to-end workflow tests ``` ### Test Coverage The project maintains comprehensive test coverage including: - **Activity Tests**: Unit tests for all activity classes - **Workflow Tests**: Integration tests for workflow orchestration - **Utility Tests**: Tests for configuration builders and converters - **Database Tests**: Tests for MongoDB, Redis, and PostgreSQL operations - **Email Tests**: Tests for email generation and delivery - **Notification Tests**: Tests for filtering and caching logic ### Test Execution ```bash # Install test dependencies pip install -r requirements-dev.txt # Run unit tests with coverage (default testpaths=tests; E2E excluded) pytest --cov=orchestrator --cov-report=html # Run specific test modules pytest tests/orchestrator/activities/test_mongo_db.py pytest tests/orchestrator/workflows/test_orchestrator.py ``` ### End-to-end tests E2E tests live in `e2e/` and require **Docker** (testcontainers). They are **not** collected by default `pytest` at the repo root. ```bash source ./venv/bin/activate pip install -r requirements-dev.txt pytest e2e/ --override-ini testpaths=e2e -m e2e -v ``` Coverage is kept separate from unit tests: set `COVERAGE_FILE=.coverage.e2e` when measuring E2E coverage (see `e2e/README.md`). ## πŸ›‘οΈ Code Quality & Validation ### Overview Since Python is not compiled, this project ships a validation workflow to catch issues early. Use the `validate.sh` script to run formatting, linting, type checks, security analysis, and tests in one command. ### Validation Tools - **Ruff**: formatting and linting (fast, replaces Black/Flake8) - **mypy**: static typing checks - **Bandit**: security static analysis - **pytest**: unit/integration tests with coverage ### Tools Installation ```bash pip install -r requirements-dev.txt ``` ### Complete Validation (recommended) ```bash ./validate.sh ``` What `validate.sh` does: 1. Checks formatting with Ruff 2. Lints code with Ruff 3. Runs mypy type checking 4. Runs Bandit security analysis 5. Executes pytest with coverage (generates HTML report) Exit codes are propagated so CI can fail fast when quality gates are not met. ### Individual Commands ```bash # 1) Format check ruff format --check orchestrator/ tests/ # 2) Lint ruff check orchestrator/ tests/ # 3) Type check mypy orchestrator/ # 4) Security bandit -r orchestrator/ -ll # 5) Tests with coverage pytest tests/ --cov=orchestrator --cov-report=html ``` ### Automatic Fixes ```bash # Apply formatting ruff format orchestrator/ tests/ # Autofix common lint issues ruff check --fix orchestrator/ tests/ ``` ### Configuration Tooling is configured in `pyproject.toml` (lint rules, formatting, typing). Adjust thresholds and rules there as needed. ## πŸ”§ Development ### Project Structure ``` orchestrator/ β”œβ”€β”€ activities/ # Temporal activity implementations β”‚ β”œβ”€β”€ activities.py # Main activities orchestrator β”‚ β”œβ”€β”€ temporal_manager.py # Temporal schedule operations β”‚ β”œβ”€β”€ slot_manager.py # Redis slot management and notification filtering β”‚ β”œβ”€β”€ mongo_db.py # MongoDB operations β”‚ β”œβ”€β”€ email.py # Email service operations β”‚ β”œβ”€β”€ formatters.py # Configuration formatting and report filtering β”œβ”€β”€ workflows/ # Temporal workflow definitions β”‚ β”œβ”€β”€ orchestrator.py # Main orchestration workflow β”‚ β”œβ”€β”€ alerts.py # Error alert workflow β”‚ β”œβ”€β”€ reports.py # Scheduled report workflow β”‚ └── subworkflows/ # Sub-workflow implementations β”‚ β”œβ”€β”€ load_notification_package.py # Notification data loading β”‚ └── process_notifications.py # Email processing and delivery β”œβ”€β”€ worker/ # Worker implementation β”‚ └── worker.py # Main worker orchestrator β”œβ”€β”€ utils/ # Utility functions β”‚ β”œβ”€β”€ connectors_config.py # Database configuration β”‚ β”œβ”€β”€ email_builder.py # Email template generation β”‚ β”œβ”€β”€ orchestrator_functions.py # Pipeline utilities β”‚ β”œβ”€β”€ converters.py # Data type conversion utilities β”‚ └── templates/ # Email HTML templates β”‚ β”œβ”€β”€ email_template.html # Report email template β”‚ └── general_template.html # General email template └── metrics.py # Prometheus metrics definitions ``` ### Adding New Features 1. **Follow Temporal patterns** for new workflows and activities 2. **Add comprehensive docstrings** for all public methods 3. **Include Prometheus metrics** for monitoring 4. **Add unit tests** for new functionality 5. **Update this README** with new features and configuration ## πŸ› Troubleshooting ### Common Issues 1. **Temporal Connection Failures** - Verify Temporal server is running and accessible - Check namespace configuration and permissions - Review server logs for connection issues 2. **Database Connection Issues** - Verify all database services are running - Check connection credentials and network access - Ensure proper connection pool configuration 3. **Email Delivery Failures** - Verify SMTP server configuration and credentials - Check email sender permissions and authentication - Review email delivery logs for specific errors 4. **Workflow Execution Failures** - Review activity error logs and notifications - Check MongoDB collection configurations - Verify input data format and required fields 5. **E2E / testcontainers leftovers** - If a run is interrupted, containers may keep running. List and remove them: ```bash docker ps -a --filter label=org.testcontainers=true docker rm -f $(docker ps -aq --filter label=org.testcontainers=true) ``` ### Debug Mode Enable debug logging by setting the log level: ```bash export LOG_LEVEL=DEBUG ``` ## ⚑ Performance Tuning ### Key Parameters - **Worker Concurrency**: Configure worker task limits in Temporal client - **Connection Pools**: Optimize database connection pool sizes - **Redis TTL**: Adjust cache TTL settings based on requirements - **Batch Sizes**: Configure notification processing batch sizes ### Scaling Considerations - **Horizontal Scaling**: Deploy multiple worker instances - **Task Queue Distribution**: Use dedicated queues for different workflows - **Database Performance**: Optimize indexes and connection pooling - **Memory Management**: Monitor and configure appropriate resource limits ## 🀝 Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes with comprehensive testing 4. Update documentation and docstrings 5. Submit a pull request ### Code Quality Standards - Follow PEP 8 style guidelines - Include comprehensive docstrings for all public methods - Maintain test coverage above 80% - Use type hints where appropriate - Follow Temporal.io best practices ## πŸ“„ License This project is licensed under the terms specified in the LICENSE file. ## πŸ†˜ Support For support and questions: - Check the troubleshooting section above - Review the metrics and logs for error patterns - Open an issue in the project repository - Contact the development team --- ### Runtime queues and migration - **Queue naming**: Downstream pipeline schedules target `--queue` (for example `scouter-legacy-queue`). The orchestrator's own workers register on `orchestrator--queue`, `alerts--queue`, and `reports--queue`. - **Pipeline `runtime`**: Each pipeline JSON may include `runtime` (default `legacy` via `common_config`). Schedules without `runtime` use `legacy`. - **Schedule updates**: Changing a pipeline's `runtime` does not update an existing schedule's `task_queue` in place; the next orchestrator tick deletes and recreates the schedule on the new queue. - **Rollout order**: Deploy scouter/laborious worker fleets bound to the new `--queue` family before redeploying the orchestrator. On the first tick after upgrade, orphan schedules on old queue names are normalized away and schedules are recreated on the new queues (no in-place rename). **Note**: The SIENTIA DataOps Orchestrator is designed for production use in enterprise data environments. Ensure proper security configuration and network isolation for production deployments.