SIENTIAPDE-1646

Refactor worker task queue management and update README

- Introduced runtime-scoped task queues for workflows, replacing legacy queue names.
- Updated worker implementation to utilize `sientia_do.temporal.worker.prepare_worker`.
- Added `RUNTIME` environment variable to configure task queue suffixes.
- Enhanced README documentation to reflect changes in task queue structure and worker setup.
This commit is contained in:
vitor-aignosi
2026-05-19 17:07:18 -03:00
parent d856150e24
commit 2ccda3e440
5 changed files with 64 additions and 92 deletions

View File

@@ -126,10 +126,15 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
### Key Components
#### **Worker (`laborious/worker/worker.py`)**
- Temporal client setup, worker lifecycle, task queues
- Temporal client setup, four workers via `sientia_do.temporal.worker.prepare_worker`
- Runtime-scoped task queues: `{workflow}-{RUNTIME}-queue` for all workflows
- Metrics server initialization, notification handler setup
- Graceful shutdown and autoscaling-friendly behavior
**Breaking (schedulers):** drift and simple_metrics queues are no longer `drift-queue` /
`simple_metrics-queue`. Use `drift-{RUNTIME}-queue` and `simple_metrics-{RUNTIME}-queue`
matching the worker pod `RUNTIME` env (same as `predictions_batch` / `minimal_retrain`).
#### **Workflows (`laborious/workflows/`)**
- `predictions_batch.py`: Batch prediction entry point
- `sub_workflows/prediction_process.py`: Core prediction pipeline
@@ -802,6 +807,7 @@ See [OPC UA Communication](#opc-ua-communication) for semantics, concurrency, an
|----------|-------------|---------|----------|
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
| `TEMPORAL_NAMESPACE` | Temporal namespace | `laborious` | No |
| `RUNTIME` | Task queue suffix for all workflows (`{workflow}-{RUNTIME}-queue`) | _(none)_ | Yes |
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
@@ -1097,8 +1103,7 @@ laborious/
│ ├── prediction_process.py # Core prediction workflow
│ └── format_and_export_prediction.py # Export workflow
├── worker/ # Worker implementation
── worker.py # Main worker orchestrator
│ └── prepare_worker.py # Worker factory with autoscaling config
── worker.py # Main worker orchestrator (uses sientia_do prepare_worker)
├── utils/ # Utility functions
│ ├── connectors_config.py # Environment-driven config builders
│ ├── models/ # Data models
@@ -1179,7 +1184,9 @@ export LOG_LEVEL=DEBUG
### Scaling Considerations
- **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Use multiple task queues for different workflow types
- **Task Queue Distribution**: One worker pod per `RUNTIME`; queues are
`predictions_batch-{RUNTIME}-queue`, `minimal_retrain-{RUNTIME}-queue`,
`drift-{RUNTIME}-queue`, `simple_metrics-{RUNTIME}-queue`
- **Database Performance**: Optimize indexes and connection pooling
- **MLFlow Performance**: Configure appropriate model serving resources