SIENTIAPDE-1478

Update tests.ipynb and prepare_worker.py

- Reset execution counts in tests.ipynb for reproducibility.
- Removed error outputs and added new tags in TAG_NAMES for enhanced data retrieval.
- Introduced JSON export of web_ids to 'web_ids.json' for better data management.
- Added comments in prepare_worker.py to clarify worker configuration parameters.

scouter/worker/prepare_worker.py

@@ -7,6 +7,8 @@ from sientia_do.observability.logger import Logger
 from temporalio.client import Client
 from temporalio.worker import PollerBehaviorAutoscaling, Worker
 
+# Worker configuration parameters with default values
+# See worker_parameters.md for detailed documentation
 parameters = [
     ('MAX_CONCURRENT_WORKFLOW_TASKS', '200'),
     ('MAX_CONCURRENT_ACTIVITIES', '200'),

scouter/worker/worker_parameters.md

@@ -0,0 +1,113 @@
+# Worker Parameters Documentation
+
+This document explains each configuration parameter used in the `prepare_worker.py` file for configuring Temporal workers.
+
+## Overview
+
+All parameters can be configured via environment variables using the pattern: `{WORKFLOW_NAME}_{PARAMETER_NAME}`. If not set, default values are used as specified below.
+
+## Concurrency Parameters
+
+### MAX_CONCURRENT_WORKFLOW_TASKS
+- **Default**: `200`
+- **Description**: Maximum number of concurrent workflow tasks that can be processed simultaneously by the worker. This controls how many workflow executions can be actively running at the same time.
+- **Usage**: Set via `max_concurrent_workflow_tasks` in the Worker configuration.
+- **Impact**: Higher values allow more workflows to run concurrently but consume more resources. Lower values provide better resource control but may limit throughput.
+
+### MAX_CONCURRENT_ACTIVITIES
+- **Default**: `200`
+- **Description**: Maximum number of concurrent activity tasks that can be executed simultaneously by the worker. Activities are the actual work units that perform business logic.
+- **Usage**: Set via `max_concurrent_activities` in the Worker configuration.
+- **Impact**: Controls the parallelism of activity execution. Higher values increase throughput but require more system resources (CPU, memory, network connections).
+
+### MAX_CONCURRENT_LOCAL_ACTIVITIES
+- **Default**: `200`
+- **Description**: Maximum number of concurrent local activity tasks that can be executed simultaneously. Local activities run in the same process as the workflow, without requiring a separate activity worker.
+- **Usage**: Set via `max_concurrent_local_activities` in the Worker configuration.
+- **Impact**: Similar to regular activities, but local activities have lower latency and overhead since they don't require network round-trips. Useful for lightweight operations.
+
+## Caching Parameters
+
+### MAX_CACHED_WORKFLOWS
+- **Default**: `200`
+- **Description**: Maximum number of workflow instances that can be cached in memory by the worker. Cached workflows allow faster resumption of execution without reloading state.
+- **Usage**: Set via `max_cached_workflows` in the Worker configuration.
+- **Impact**: Higher values improve performance for frequently accessed workflows but consume more memory. Lower values reduce memory usage but may require more frequent state reloads.
+
+## Understanding Pollers in Temporal
+
+**Pollers** are components of Temporal Workers that continuously request tasks from the Temporal service's Task Queues via synchronous RPCs. There are separate pollers for workflow tasks and activity tasks.
+
+### How Pollers Work
+
+Pollers send requests to the Temporal service to retrieve tasks from Task Queues. When a task is available, the poller retrieves it and the Worker processes it using registered Workflow or Activity handlers. This architecture provides:
+- **Load Balancing**: Workers only poll when they have capacity, distributing load across multiple processes
+- **Fault Tolerance**: Tasks persist in queues if a Worker fails, allowing recovery
+- **Task Routing**: Tasks can be routed to specific Worker processes
+
+### Autoscaling Poller Behavior
+
+Temporal supports autoscaling that dynamically adjusts the number of concurrent pollers based on workload. The system scales up during high load and down during low load, maintaining a baseline for responsiveness. Autoscaling is configured with `minimum`, `initial`, and `maximum` parameters that define the scaling bounds.
+
+## Workflow Poller Behavior (Autoscaling)
+
+These parameters control the autoscaling behavior of the workflow task poller, which retrieves workflow tasks from the Temporal server.
+
+### WORKFLOW_POLLER_BEHAVIUR_MINIMUM
+- **Default**: `10`
+- **Description**: Minimum number of concurrent pollers for workflow tasks. The poller count will never go below this value.
+- **Usage**: Set via `minimum` in `PollerBehaviorAutoscaling` for `workflow_task_poller_behavior`.
+- **Impact**: Ensures a baseline level of polling activity even during low load periods.
+
+### WORKFLOW_POLLER_BEHAVIUR_INITIAL
+- **Default**: `100`
+- **Description**: Initial number of concurrent pollers for workflow tasks when the worker starts.
+- **Usage**: Set via `initial` in `PollerBehaviorAutoscaling` for `workflow_task_poller_behavior`.
+- **Impact**: Determines the starting point for poller scaling. Higher values provide faster initial task acquisition but consume more resources.
+
+### WORKFLOW_POLLER_BEHAVIUR_MAXIMUM
+- **Default**: `200`
+- **Description**: Maximum number of concurrent pollers allowed for workflow tasks. The poller count will not exceed this value even under high load.
+- **Usage**: Set via `maximum` in `PollerBehaviorAutoscaling` for `workflow_task_poller_behavior`.
+- **Impact**: Caps the resource consumption for workflow task polling. Prevents excessive polling that could overwhelm the Temporal server or worker.
+
+## Activity Poller Behavior (Autoscaling)
+
+These parameters control the autoscaling behavior of the activity task poller, which retrieves activity tasks from the Temporal server.
+
+### ACTIVITY_POLLER_BEHAVIUR_MINIMUM
+- **Default**: `10`
+- **Description**: Minimum number of concurrent pollers for activity tasks. The poller count will never go below this value.
+- **Usage**: Set via `minimum` in `PollerBehaviorAutoscaling` for `activity_task_poller_behavior`.
+- **Impact**: Ensures a baseline level of polling activity even during low load periods.
+
+### ACTIVITY_POLLER_BEHAVIUR_INITIAL
+- **Default**: `100`
+- **Description**: Initial number of concurrent pollers for activity tasks when the worker starts.
+- **Usage**: Set via `initial` in `PollerBehaviorAutoscaling` for `activity_task_poller_behavior`.
+- **Impact**: Determines the starting point for poller scaling. Higher values provide faster initial task acquisition but consume more resources.
+
+### ACTIVITY_POLLER_BEHAVIUR_MAXIMUM
+- **Default**: `200`
+- **Description**: Maximum number of concurrent pollers allowed for activity tasks. The poller count will not exceed this value even under high load.
+- **Usage**: Set via `maximum` in `PollerBehaviorAutoscaling` for `activity_task_poller_behavior`.
+- **Impact**: Caps the resource consumption for activity task polling. Prevents excessive polling that could overwhelm the Temporal server or worker.
+
+## Configuration Example
+
+To override these parameters, set environment variables using the pattern:
+```
+{WORKFLOW_NAME}_{PARAMETER_NAME}={value}
+```
+
+For example, if your workflow is named `ScouterWorkflow`:
+```bash
+SCOUTERWORKFLOW_MAX_CONCURRENT_ACTIVITIES=500
+SCOUTERWORKFLOW_WORKFLOW_POLLER_BEHAVIUR_MAXIMUM=300
+```
+
+## Notes
+
+- All parameter values are converted to integers before use.
+- The autoscaling poller behavior dynamically adjusts the number of pollers between the minimum and maximum values based on workload.
+- These parameters should be tuned based on your specific workload characteristics, available resources, and performance requirements.