SIENTIAPDE-1646

Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.

README.md

@@ -390,16 +390,17 @@ The orchestrator includes an advanced notification filtering system that prevent
 - **Purpose**: Main application orchestrator managing Temporal workers and task queues
 - **Responsibilities**:
   - Temporal client initialization and connection management with SDK metrics
-  - Worker lifecycle management and graceful shutdown
-  - Task queue configuration (orchestrator, alerts, reports) with dedicated workers
+  - Worker lifecycle management and graceful shutdown via `sientia_do.temporal.worker.prepare_worker`
+  - Task queue configuration (orchestrator, alerts, reports) with dedicated workers on `<workflow>-<runtime>-queue`
   - Prometheus metrics server initialization on HTTP_METRICS_PORT
   - Temporal SDK metrics server initialization on HTTP_SDK_METRICS_PORT
   - Notification handler setup and configuration
 - **Key Features**:
   - Multi-queue worker management with three dedicated workers (orchestrator, alerts, reports)
+  - `RUNTIME` env var (default `legacy`) passed to every `prepare_worker` call
+  - Sync blocking activities run on `prepare_worker`'s `activity_executor` thread pool; workflows stay `async def`
   - Application health metrics (app_up gauge) for Kubernetes liveness/readiness probes
-  - Graceful shutdown with cleanup procedures for all connections
-  - Comprehensive error handling and metrics collection
+  - Graceful shutdown with cleanup procedures for all connections and exit code propagation to Kubernetes
   - Parallel worker execution using asyncio.gather
 
 #### **Activities (`orchestrator/activities/`)**
@@ -524,6 +525,11 @@ python -m orchestrator.worker.worker
 | `TEMPORAL_NAMESPACE` | Default Temporal namespace | `default` | No |
 | `TEMPORAL_SCOUTER_NAMESPACE` | Scouter workflow namespace | `scouter` | No |
 | `TEMPORAL_LABORIOUS_NAMESPACE` | Laborious workflow namespace | `laborious` | No |
+| `RUNTIME` | Runtime slice for orchestrator worker task queues (`orchestrator-<runtime>-queue`, etc.) | `legacy` | No |
+| `ACTIVITY_EXECUTOR_MAX_WORKERS` | Thread pool size for sync activities (all workers) | `200` | No |
+| `ORCHESTRATOR_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Orchestrator worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No |
+| `ALERTS_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Alerts worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No |
+| `REPORTS_ACTIVITY_EXECUTOR_MAX_WORKERS` | Per-worker override for the Reports worker | falls back to `ACTIVITY_EXECUTOR_MAX_WORKERS` | No |
 | `REDIS_HOST` | Redis server hostname | `localhost` | Yes |
 | `REDIS_PORT` | Redis server port | `6379` | Yes |
 | `REDIS_USERNAME` | Redis username | `default` | Yes |
@@ -673,16 +679,28 @@ The project maintains comprehensive test coverage including:
 ### Test Execution
 ```bash
 # Install test dependencies
-pip install pytest pytest-cov pytest-asyncio
+pip install -r requirements-dev.txt
 
-# Run tests with coverage
+# Run unit tests with coverage (default testpaths=tests; E2E excluded)
 pytest --cov=orchestrator --cov-report=html
 
 # Run specific test modules
-pytest tests/activities/test_mongo_db.py
-pytest tests/workflows/test_orchestrator.py
+pytest tests/orchestrator/activities/test_mongo_db.py
+pytest tests/orchestrator/workflows/test_orchestrator.py
 ```
 
+### End-to-end tests
+
+E2E tests live in `e2e/` and require **Docker** (testcontainers). They are **not** collected by default `pytest` at the repo root.
+
+```bash
+source ./venv/bin/activate
+pip install -r requirements-dev.txt
+pytest e2e/ --override-ini testpaths=e2e -m e2e -v
+```
+
+Coverage is kept separate from unit tests: set `COVERAGE_FILE=.coverage.e2e` when measuring E2E coverage (see `e2e/README.md`).
+
 ## 🛡️ Code Quality & Validation
 
 ### Overview
@@ -814,6 +832,13 @@ orchestrator/
    - Check MongoDB collection configurations
    - Verify input data format and required fields
 
+5. **E2E / testcontainers leftovers**
+   - If a run is interrupted, containers may keep running. List and remove them:
+   ```bash
+   docker ps -a --filter label=org.testcontainers=true
+   docker rm -f $(docker ps -aq --filter label=org.testcontainers=true)
+   ```
+
 ### Debug Mode
 
 Enable debug logging by setting the log level:
@@ -867,4 +892,11 @@ For support and questions:
 
 ---
 
+### Runtime queues and migration
+
+- **Queue naming**: Downstream pipeline schedules target `<workflow_type>-<runtime>-queue` (for example `scouter-legacy-queue`). The orchestrator's own workers register on `orchestrator-<runtime>-queue`, `alerts-<runtime>-queue`, and `reports-<runtime>-queue`.
+- **Pipeline `runtime`**: Each pipeline JSON may include `runtime` (default `legacy` via `common_config`). Schedules without `runtime` use `legacy`.
+- **Schedule updates**: Changing a pipeline's `runtime` does not update an existing schedule's `task_queue` in place; the next orchestrator tick deletes and recreates the schedule on the new queue.
+- **Rollout order**: Deploy scouter/laborious worker fleets bound to the new `<workflow_type>-<runtime>-queue` family before redeploying the orchestrator. On the first tick after upgrade, orphan schedules on old queue names are normalized away and schedules are recreated on the new queues (no in-place rename).
+
 **Note**: The SIENTIA DataOps Orchestrator is designed for production use in enterprise data environments. Ensure proper security configuration and network isolation for production deployments.