gitea_admin/sientia-dataops-scouter_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1110

Browse Source 
Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
This commit is contained in:
vitor-aignosi
2025-06-30 14:11:30 -03:00
parent 9afb74be21
commit 6144bcf8ff
2 changed files with 9 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
scouter/activities/kafka.py
View File

@@ -56,7 +56,14 @@ class Kafka(BaseActivity):
topic = input_data["topic"] topic = input_data["topic"]
# Subscribe to the specified topic # Subscribe to the specified topic
self.kafka_connector.subscribe([topic]) self.kafka_connector = KafkaConsumer(
topic,
bootstrap_servers=self.bootstrap_servers,
auto_offset_reset="earliest",
enable_auto_commit=True,
group_id=self.group_id,
value_deserializer=lambda x: json.loads(x.decode("utf-8"))
)
# List to store message values # List to store message values
message_values = [] message_values = []

2
values.yaml
View File

@@ -146,7 +146,7 @@ env:
- name: KAFKA_BOOTSTRAP_SERVERS - name: KAFKA_BOOTSTRAP_SERVERS
value: "kafka.kafka.svc.cluster.local:9092" value: "kafka.kafka.svc.cluster.local:9092"
- name: KAFKA_POLLING_TIME - name: KAFKA_POLLING_TIME
value: "1000" value: "10000"
- name: REDIS_HOST - name: REDIS_HOST
value: "redis-master.redis.svc.cluster.local" value: "redis-master.redis.svc.cluster.local"