SIENTIAPDE-1646
Refactor worker task queue management and update README - Introduced runtime-scoped task queues for workflows, replacing legacy queue names. - Updated worker implementation to utilize `sientia_do.temporal.worker.prepare_worker`. - Added `RUNTIME` environment variable to configure task queue suffixes. - Enhanced README documentation to reflect changes in task queue structure and worker setup.
This commit is contained in:
15
README.md
15
README.md
@@ -126,10 +126,15 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
|
||||
### Key Components
|
||||
|
||||
#### **Worker (`laborious/worker/worker.py`)**
|
||||
- Temporal client setup, worker lifecycle, task queues
|
||||
- Temporal client setup, four workers via `sientia_do.temporal.worker.prepare_worker`
|
||||
- Runtime-scoped task queues: `{workflow}-{RUNTIME}-queue` for all workflows
|
||||
- Metrics server initialization, notification handler setup
|
||||
- Graceful shutdown and autoscaling-friendly behavior
|
||||
|
||||
**Breaking (schedulers):** drift and simple_metrics queues are no longer `drift-queue` /
|
||||
`simple_metrics-queue`. Use `drift-{RUNTIME}-queue` and `simple_metrics-{RUNTIME}-queue`
|
||||
matching the worker pod `RUNTIME` env (same as `predictions_batch` / `minimal_retrain`).
|
||||
|
||||
#### **Workflows (`laborious/workflows/`)**
|
||||
- `predictions_batch.py`: Batch prediction entry point
|
||||
- `sub_workflows/prediction_process.py`: Core prediction pipeline
|
||||
@@ -802,6 +807,7 @@ See [OPC UA Communication](#opc-ua-communication) for semantics, concurrency, an
|
||||
|----------|-------------|---------|----------|
|
||||
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
|
||||
| `TEMPORAL_NAMESPACE` | Temporal namespace | `laborious` | No |
|
||||
| `RUNTIME` | Task queue suffix for all workflows (`{workflow}-{RUNTIME}-queue`) | _(none)_ | Yes |
|
||||
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
|
||||
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
|
||||
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
|
||||
@@ -1097,8 +1103,7 @@ laborious/
|
||||
│ ├── prediction_process.py # Core prediction workflow
|
||||
│ └── format_and_export_prediction.py # Export workflow
|
||||
├── worker/ # Worker implementation
|
||||
│ ├── worker.py # Main worker orchestrator
|
||||
│ └── prepare_worker.py # Worker factory with autoscaling config
|
||||
│ └── worker.py # Main worker orchestrator (uses sientia_do prepare_worker)
|
||||
├── utils/ # Utility functions
|
||||
│ ├── connectors_config.py # Environment-driven config builders
|
||||
│ ├── models/ # Data models
|
||||
@@ -1179,7 +1184,9 @@ export LOG_LEVEL=DEBUG
|
||||
### Scaling Considerations
|
||||
|
||||
- **Horizontal Scaling**: Deploy multiple worker instances
|
||||
- **Task Queue Distribution**: Use multiple task queues for different workflow types
|
||||
- **Task Queue Distribution**: One worker pod per `RUNTIME`; queues are
|
||||
`predictions_batch-{RUNTIME}-queue`, `minimal_retrain-{RUNTIME}-queue`,
|
||||
`drift-{RUNTIME}-queue`, `simple_metrics-{RUNTIME}-queue`
|
||||
- **Database Performance**: Optimize indexes and connection pooling
|
||||
- **MLFlow Performance**: Configure appropriate model serving resources
|
||||
|
||||
|
||||
Reference in New Issue
Block a user