SIENTIAPDE-988

Enhance ingestor and manager classes with notification handling

- Integrated NotificationHandler into Ingestor, DataManager, IngestorManager, and OpcManager for improved error reporting and monitoring.
- Updated methods to send notifications on critical events such as Kafka publishing errors, OPC connection issues, and cycle count warnings.
- Refactored related tests to ensure coverage of new notification functionalities and validate integration with existing components.
- Improved logging and error handling across the system to enhance traceability and operational insights.
This commit is contained in:
vitor-aignosi
2025-04-30 19:28:43 -03:00
parent 3fc3753636
commit d00078d323
8 changed files with 423 additions and 168 deletions

View File

@@ -2,6 +2,7 @@ from logging import Formatter, StreamHandler, getLogger
from os import getenv
from ingestor.managers.ingestor_manager import IngestorManager
from sientia_do.notifications.handlers import NotificationHandler
class Ingestor:
@@ -37,14 +38,27 @@ class Ingestor:
self.poll_interval = int(getenv("POLL_INTERVAL", 5))
self.kafka_servers = kafka_servers.split(",")
self.logger = None
self.init_logger()
self.notification_handler = NotificationHandler(
servers=self.kafka_servers,
logger=self.logger,
project_name="OPC_INGESTOR",
pipeline_name="-",
trigger_name="-",
model_name="-",
model="-"
)
# build args for build notificarions components
# call build notifications components
def init_logger(self):
"""
Initializes a logger instance for the class.
This method sets up a logger with a specified log level, a stream handler,
and a formatter. The log level is determined by the environment variable
"LOG_LEVEL", defaulting to "INFO" if not set. The logger is then attached
This method sets up a logger with a specified log level, a stream handler,
and a formatter. The log level is determined by the environment variable
"LOG_LEVEL", defaulting to "INFO" if not set. The logger is then attached
to the instance for use throughout the class.
Attributes:
self.logger (logging.Logger): The configured logger instance.
@@ -105,7 +119,8 @@ class Ingestor:
self.ingestor_manager = IngestorManager(
self.kafka_servers, self.redis_host, self.redis_port, self.lease_ttl,
self.heartbeat_ttl, self.pod_id, self.poll_interval, self.logger, self.redis_username, self.redis_password
self.heartbeat_ttl, self.pod_id, self.poll_interval, self.logger,
self.notification_handler, self.redis_username, self.redis_password
)
# Declare ingestor ative
@@ -120,9 +135,9 @@ class Ingestor:
def manage_no_slots(self, number_of_slots: int):
"""
Manages the scenario where there are no slots assigned to the ingestor.
This method checks if the ingestor is active (i.e., has no managed tags)
and if the number of available slots is greater than zero. If both
conditions are met, it attempts to acquire a slot lease and handles
This method checks if the ingestor is active (i.e., has no managed tags)
and if the number of available slots is greater than zero. If both
conditions are met, it attempts to acquire a slot lease and handles
the acquired tags accordingly.
Args:
number_of_slots (int): The number of available slots.
@@ -138,16 +153,16 @@ class Ingestor:
def manage_leases(self, available_slots: int, lacking_ingestors: int, slot_diff: int):
"""
Manages the allocation and deallocation of slot leases for ingestors based on
Manages the allocation and deallocation of slot leases for ingestors based on
the number of available slots, lacking ingestors, and slot differences.
Args:
available_slots (int): The number of slots currently available for allocation.
lacking_ingestors (int): The number of ingestors that are active and without slots.
slot_diff (int): The difference between the total slots and the required slots.
Behavior:
- If there are available slots and lacking ingestors, attempts to acquire slot leases
- If there are available slots and lacking ingestors, attempts to acquire slot leases
for the available slots and processes the acquired tags.
- If there are no lacking ingestors but there are extra slots (slot_diff > 0),
- If there are no lacking ingestors but there are extra slots (slot_diff > 0),
releases the extra slot leases to ensure proper allocation.
Logs:
- Logs the number of available slots when attempting to acquire leases.
@@ -223,3 +238,6 @@ class Ingestor:
# Update opc servers
self.ingestor_manager.update_slot_config()
# Check OPC cycles
self.ingestor_manager.check_opc_servers_integrity()