sientia-dataops-scouter_temporal/scouter/worker/worker_parameters.md

Worker Parameters Documentation

This document explains each configuration parameter used in the prepare_worker.py file for configuring Temporal workers.

Overview

All parameters can be configured via environment variables using the pattern: {WORKFLOW_NAME}_{PARAMETER_NAME}. If not set, default values are used as specified below.

Concurrency Parameters

MAX_CONCURRENT_WORKFLOW_TASKS

• Default: 200
• Description: Maximum number of concurrent workflow tasks that can be processed simultaneously by the worker. This controls how many workflow executions can be actively running at the same time.
• Usage: Set via max_concurrent_workflow_tasks in the Worker configuration.
• Impact: Higher values allow more workflows to run concurrently but consume more resources. Lower values provide better resource control but may limit throughput.

MAX_CONCURRENT_ACTIVITIES

• Default: 200
• Description: Maximum number of concurrent activity tasks that can be executed simultaneously by the worker. Activities are the actual work units that perform business logic.
• Usage: Set via max_concurrent_activities in the Worker configuration.
• Impact: Controls the parallelism of activity execution. Higher values increase throughput but require more system resources (CPU, memory, network connections).

MAX_CONCURRENT_LOCAL_ACTIVITIES

• Default: 200
• Description: Maximum number of concurrent local activity tasks that can be executed simultaneously. Local activities run in the same process as the workflow, without requiring a separate activity worker.
• Usage: Set via max_concurrent_local_activities in the Worker configuration.
• Impact: Similar to regular activities, but local activities have lower latency and overhead since they don't require network round-trips. Useful for lightweight operations.

Caching Parameters

MAX_CACHED_WORKFLOWS

• Default: 200
• Description: Maximum number of workflow instances that can be cached in memory by the worker. Cached workflows allow faster resumption of execution without reloading state.
• Usage: Set via max_cached_workflows in the Worker configuration.
• Impact: Higher values improve performance for frequently accessed workflows but consume more memory. Lower values reduce memory usage but may require more frequent state reloads.

Understanding Pollers in Temporal

Pollers are components of Temporal Workers that continuously request tasks from the Temporal service's Task Queues via synchronous RPCs. There are separate pollers for workflow tasks and activity tasks.

How Pollers Work

Pollers send requests to the Temporal service to retrieve tasks from Task Queues. When a task is available, the poller retrieves it and the Worker processes it using registered Workflow or Activity handlers. This architecture provides:

• Load Balancing: Workers only poll when they have capacity, distributing load across multiple processes
• Fault Tolerance: Tasks persist in queues if a Worker fails, allowing recovery
• Task Routing: Tasks can be routed to specific Worker processes

Autoscaling Poller Behavior

Temporal supports autoscaling that dynamically adjusts the number of concurrent pollers based on workload. The system scales up during high load and down during low load, maintaining a baseline for responsiveness. Autoscaling is configured with minimum, initial, and maximum parameters that define the scaling bounds.

Workflow Poller Behavior (Autoscaling)

These parameters control the autoscaling behavior of the workflow task poller, which retrieves workflow tasks from the Temporal server.

WORKFLOW_POLLER_BEHAVIOUR_MINIMUM

• Default: 10
• Description: Minimum number of concurrent pollers for workflow tasks. The poller count will never go below this value.
• Usage: Set via minimum in PollerBehaviorAutoscaling for workflow_task_poller_behavior.
• Impact: Ensures a baseline level of polling activity even during low load periods.

WORKFLOW_POLLER_BEHAVIOUR_INITIAL

• Default: 100
• Description: Initial number of concurrent pollers for workflow tasks when the worker starts.
• Usage: Set via initial in PollerBehaviorAutoscaling for workflow_task_poller_behavior.
• Impact: Determines the starting point for poller scaling. Higher values provide faster initial task acquisition but consume more resources.

WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM

• Default: 200
• Description: Maximum number of concurrent pollers allowed for workflow tasks. The poller count will not exceed this value even under high load.
• Usage: Set via maximum in PollerBehaviorAutoscaling for workflow_task_poller_behavior.
• Impact: Caps the resource consumption for workflow task polling. Prevents excessive polling that could overwhelm the Temporal server or worker.

Activity Poller Behavior (Autoscaling)

These parameters control the autoscaling behavior of the activity task poller, which retrieves activity tasks from the Temporal server.

ACTIVITY_POLLER_BEHAVIOUR_MINIMUM

• Default: 10
• Description: Minimum number of concurrent pollers for activity tasks. The poller count will never go below this value.
• Usage: Set via minimum in PollerBehaviorAutoscaling for activity_task_poller_behavior.
• Impact: Ensures a baseline level of polling activity even during low load periods.

ACTIVITY_POLLER_BEHAVIOUR_INITIAL

• Default: 100
• Description: Initial number of concurrent pollers for activity tasks when the worker starts.
• Usage: Set via initial in PollerBehaviorAutoscaling for activity_task_poller_behavior.
• Impact: Determines the starting point for poller scaling. Higher values provide faster initial task acquisition but consume more resources.

ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM

• Default: 200
• Description: Maximum number of concurrent pollers allowed for activity tasks. The poller count will not exceed this value even under high load.
• Usage: Set via maximum in PollerBehaviorAutoscaling for activity_task_poller_behavior.
• Impact: Caps the resource consumption for activity task polling. Prevents excessive polling that could overwhelm the Temporal server or worker.

Configuration Example

To override these parameters, set environment variables using the pattern:

{WORKFLOW_NAME}_{PARAMETER_NAME}={value}

For example, if your workflow is named ScouterWorkflow:

SCOUTERWORKFLOW_MAX_CONCURRENT_ACTIVITIES=500
SCOUTERWORKFLOW_WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM=300

Notes

• All parameter values are converted to integers before use.
• The autoscaling poller behavior dynamically adjusts the number of pollers between the minimum and maximum values based on workload.
• These parameters should be tuned based on your specific workload characteristics, available resources, and performance requirements.