SIENTIAPDE-1273

Enhance resource management and configuration in Laborious worker

- Updated `values.yaml` to define resource limits and requests for better performance tuning.
- Modified environment variables in `worker.py` to support resource-based scaling and improved task queue management.
- Introduced new functions for creating resource tuners and poller behaviors, enhancing scalability and efficiency in handling workloads.

values.yaml

@@ -52,17 +52,15 @@ securityContext: {}
   # runAsUser: 1000
 
 
-resources: {}
-  # We usually recommend not to specify default resources and to leave this as a conscious
-  # choice for the user. This also increases chances charts run on environments with little
-  # resources, such as Minikube. If you do want to specify resources, uncomment the following
-  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
-  # limits:
-  #   cpu: 100m
-  #   memory: 128Mi
-  # requests:
-  #   cpu: 100m
-  #   memory: 128Mi
+resources:
+  # Resource limits and requests are important for ResourceBasedTuner to work correctly.
+  # The tuner monitors system CPU and memory usage, so proper resource limits must be set.
+  limits:
+    cpu: 2000m  # 2 CPU cores
+    memory: 20Gi  # 20 GB memory
+  requests:
+    cpu: 1000m  # 1 CPU core
+    memory: 2Gi  # 2 GB memory
 
 # This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
 livenessProbe:
@@ -167,9 +165,11 @@ env:
   - name: POSTGRES_DBNAME
     value: "sientia"
   - name: POSTGRES_MIN_CONNECTIONS
-    value: "10"
+    value: "20"
+  # max_connections = number_of_workers * max_concurrent_activities * safety_factor
+  # Example: 4 workers * 50 activities * 0.5 = 100 connections
   - name: POSTGRES_MAX_CONNECTIONS
-    value: "30"
+    value: "100"
 
   - name: MLFLOW_HOST
     value: "http://sientia-tracker-mlflow-tracking.sientia-tracker.svc.cluster.local"