SIENTIAPDE-1350: Integrate cleanup workflow and update default task queue names.

This commit is contained in:
Bruno Domingues
2025-11-26 22:11:20 -03:00
parent 11aaa0780f
commit 5dcb62ef4b
4 changed files with 21 additions and 12 deletions

View File

@@ -2,9 +2,11 @@
This module provides the main worker implementation for the Sientia DataOps Model Manager system.
It orchestrates Temporal workers, manages task queues, and handles the lifecycle of
model training workflows.
model training and cleanup workflows.
The worker supports the train_model-queue task queue for ML model training workflows.
The worker supports two task queues:
- train_model-queue: For ML model training workflows
- cleanup-queue: For file cleanup workflows
Key Features:
- Automatic scaling with PollerBehaviorAutoscaling
@@ -12,14 +14,18 @@ Key Features:
- Comprehensive error handling and logging
- Graceful shutdown with cleanup
- ML model training pipeline orchestration
- Automated cleanup schedule management
Environment Variables:
- TEMPORAL_HOST: Temporal server address (default: localhost:7233)
- TEMPORAL_NAMESPACE: Temporal namespace (default: model_manager)
- TEMPORAL_NAMESPACE: Temporal namespace (default: model-manager)
- TEMPORAL_USE_TLS: Enable TLS for Temporal connection (default: false)
- TRAIN_TASK_QUEUE: Task queue for training workflows (default: train_model-queue)
- CLEANUP_TASK_QUEUE: Task queue for cleanup workflows (default: cleanup-queue)
- POD_ID: Kubernetes pod identifier for metrics
- HTTP_METRICS_PORT: Prometheus metrics server port (default: 9090)
- HTTP_SDK_METRICS_PORT: Temporal SDK metrics port (default: 9091)
- PROJECT_NAME: Project name for notifications (default: model_manager)
- PROJECT_NAME: Project name for notifications (default: model-manager)
"""
from temporalio import client, workflow