SIENTIAPDE-1084
Remove deprecated files and enhance documentation - Deleted `coverage.sh`, `docker-compose.yaml`, `Dockerfile`, and simulator-related files to streamline the project structure. - Updated `README.md` to provide a comprehensive overview of the OPC Ingestor, including features, architecture, installation, usage, and troubleshooting. - Enhanced docstrings across various classes and methods in the `ingestor` module for better clarity and maintainability. - Improved Prometheus metrics documentation in `metrics.py` to ensure proper monitoring and observability of the system.
This commit is contained in:
vitor-aignosi
@@ -12,30 +12,65 @@ import ingestor.metrics as metrics
|
||||
|
||||
|
||||
class Ingestor:
|
||||
"""
|
||||
Main OPC Ingestor class that orchestrates data collection from OPC UA servers.
|
||||
|
||||
The Ingestor is responsible for:
|
||||
- Managing slot leases for load balancing across multiple instances
|
||||
- Connecting to and monitoring OPC UA servers
|
||||
- Subscribing to OPC tags and collecting real-time data
|
||||
- Distributing data to Kafka and MongoDB
|
||||
- Providing health monitoring and metrics collection
|
||||
|
||||
The ingestor uses a slot-based architecture where each slot represents
|
||||
a collection of OPC tags that can be managed by a single ingestor instance.
|
||||
This allows for horizontal scaling and load distribution.
|
||||
|
||||
Environment Variables:
|
||||
KAFKA_SERVERS: Comma-separated list of Kafka server addresses (default: "localhost:9092")
|
||||
EXPORT_TO_KAFKA: Enable/disable Kafka export (default: "false")
|
||||
REDIS_HOST: Redis server hostname (default: "localhost")
|
||||
REDIS_PORT: Redis server port (default: 6379)
|
||||
REDIS_USERNAME: Redis username (optional)
|
||||
REDIS_PASSWORD: Redis password (optional)
|
||||
LEASE_TTL: Time-to-live for slot leases in seconds (default: 10)
|
||||
HEARTBEAT_TTL: Time-to-live for heartbeats in seconds (default: 20)
|
||||
HOSTNAME: Pod identifier (default: "localhost")
|
||||
POLL_INTERVAL: Main loop polling interval in seconds (default: 5)
|
||||
MONGODB_URL: MongoDB server address (default: "localhost:27017")
|
||||
MONGODB_USERNAME: MongoDB username (default: "sientia")
|
||||
MONGODB_PASSWORD: MongoDB password (default: "sientia")
|
||||
MONGODB_DATABASE: MongoDB database name (default: "sientia")
|
||||
|
||||
Attributes:
|
||||
export_to_kafka (bool): Whether to export data to Kafka
|
||||
redis_host (str): Redis server hostname
|
||||
redis_port (int): Redis server port
|
||||
redis_username (str): Redis username (optional)
|
||||
redis_password (str): Redis password (optional)
|
||||
lease_ttl (int): Time-to-live for slot leases in seconds
|
||||
heartbeat_ttl (int): Time-to-live for heartbeat signals in seconds
|
||||
pod_id (str): Identifier for the current pod or host
|
||||
poll_interval (int): Interval in seconds for polling operations
|
||||
mongo_database (str): MongoDB database name
|
||||
mongo_connection_string (str): Complete MongoDB connection string
|
||||
kafka_servers (list): List of Kafka server addresses
|
||||
logger: Logger instance for application logging
|
||||
notification_handler: Handler for sending notifications
|
||||
metadata (dict): Application metadata for notifications and tracking
|
||||
ingestor_manager: Manager instance for coordinating operations
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
Initializes the ingestor with configuration values retrieved from environment variables.
|
||||
Environment Variables:
|
||||
KAFKA_SERVERS (str): Comma-separated list of Kafka server addresses.
|
||||
Defaults to "localhost:9092".
|
||||
REDIS_HOST (str): Hostname of the Redis server. Defaults to "localhost".
|
||||
REDIS_PORT (int): Port number of the Redis server. Defaults to 6379.
|
||||
LEASE_TTL (int): Time-to-live for leases in seconds. Defaults to 10.
|
||||
HEARTBEAT_TTL (int): Time-to-live for heartbeats in seconds. Defaults to 20.
|
||||
HOSTNAME (str): Identifier for the current pod or host. Defaults to "localhost".
|
||||
POLL_INTERVAL (int): Interval in seconds for polling operations. Defaults to 5.
|
||||
MONGODB_URL (str): URL of the MongoDB server. Defaults to "localhost:27017".
|
||||
MONGODB_USERNAME (str): Username for the MongoDB server. Defaults to "sientia".
|
||||
MONGODB_PASSWORD (str): Password for the MongoDB server. Defaults to "sientia".
|
||||
MONGODB_DATABASE (str): Name of the MongoDB database. Defaults to "sientia".
|
||||
Attributes:
|
||||
kafka_servers (list): List of Kafka server addresses.
|
||||
redis_host (str): Hostname of the Redis server.
|
||||
redis_port (int): Port number of the Redis server.
|
||||
lease_ttl (int): Time-to-live for leases in seconds.
|
||||
heartbeat_ttl (int): Time-to-live for heartbeats in seconds.
|
||||
pod_id (str): Identifier for the current pod or host.
|
||||
poll_interval (int): Interval in seconds for polling operations.
|
||||
|
||||
Sets up all necessary connections and configurations for:
|
||||
- Kafka connectivity (if enabled)
|
||||
- Redis for slot management and coordination
|
||||
- MongoDB for data persistence and notifications
|
||||
- OPC UA server management
|
||||
- Metrics collection and monitoring
|
||||
"""
|
||||
|
||||
kafka_servers = getenv("KAFKA_SERVERS", "localhost:9092")
|
||||
@@ -80,19 +115,36 @@ class Ingestor:
|
||||
self.ingestor_manager = None
|
||||
|
||||
async def shutdown(self):
|
||||
"""
|
||||
Gracefully shuts down the ingestor and all its components.
|
||||
|
||||
This method ensures proper cleanup of:
|
||||
- OPC UA connections and subscriptions
|
||||
- Resource managers and data connections
|
||||
- Active slot leases and heartbeats
|
||||
|
||||
Should be called before application termination to prevent resource leaks.
|
||||
"""
|
||||
if self.ingestor_manager:
|
||||
await self.ingestor_manager.shutdown()
|
||||
|
||||
async def handle_acquired_tags(self, acquired):
|
||||
"""
|
||||
Handles the acquired tags by subscribing to them if available.
|
||||
This method checks if there are any acquired tags. If no tags are acquired,
|
||||
it logs a warning indicating that no slots are available. Otherwise, it
|
||||
updates the OPC servers and subscribes to the acquired tags.
|
||||
|
||||
This method processes the tags that have been allocated to this ingestor
|
||||
instance through the slot leasing system. It updates OPC server configurations
|
||||
and establishes subscriptions to the allocated tags.
|
||||
|
||||
Args:
|
||||
acquired (list): A list of acquired tags to be processed. If the list
|
||||
is empty or None, no action is taken other than logging
|
||||
a warning.
|
||||
|
||||
Behavior:
|
||||
- If no tags are acquired, logs a warning about no slots being available
|
||||
- If tags are acquired, updates OPC server configurations and subscribes
|
||||
to the allocated tags for data collection
|
||||
"""
|
||||
|
||||
if not acquired:
|
||||
@@ -105,24 +157,21 @@ class Ingestor:
|
||||
|
||||
async def prepare_ingestor(self):
|
||||
"""
|
||||
Prepares the ingestor by initializing the IngestorManager, declaring the ingestor as active,
|
||||
acquiring slot leases, and handling the acquired tags.
|
||||
Prepares the ingestor by initializing all components and acquiring initial slot leases.
|
||||
|
||||
This method performs the following steps:
|
||||
1. Initializes the `IngestorManager` with the necessary configuration parameters.
|
||||
2. Declares the ingestor as active by calling `declare_active` on the `IngestorManager`.
|
||||
3. Acquires slot leases using the `get_slot_leases` method of the `IngestorManager`.
|
||||
4. Logs the acquired slots and processes them using the `handle_acquired_tags` method.
|
||||
Attributes:
|
||||
self.kafka_servers (list): List of Kafka server addresses.
|
||||
self.redis_host (str): Redis server hostname.
|
||||
self.redis_port (int): Redis server port.
|
||||
self.lease_ttl (int): Time-to-live for slot leases.
|
||||
self.heartbeat_ttl (int): Time-to-live for heartbeat signals.
|
||||
self.pod_id (str): Identifier for the current pod.
|
||||
self.poll_interval (int): Interval for polling operations.
|
||||
self.logger (Logger): Logger instance for logging messages.
|
||||
1. Initializes the IngestorManager with all necessary configuration parameters
|
||||
2. Declares the ingestor as active in the coordination system
|
||||
3. Acquires slot leases for tag management
|
||||
4. Processes the acquired tags and establishes OPC subscriptions
|
||||
|
||||
The preparation phase is critical for establishing the ingestor's role in
|
||||
the distributed system and ensuring it can begin processing OPC data.
|
||||
|
||||
Raises:
|
||||
Exception: If any error occurs during the initialization or lease acquisition process.
|
||||
This will cause the application to exit as the ingestor cannot function
|
||||
without proper initialization.
|
||||
"""
|
||||
|
||||
self.ingestor_manager = IngestorManager(
|
||||
@@ -144,7 +193,7 @@ class Ingestor:
|
||||
export_to_kafka=self.export_to_kafka,
|
||||
)
|
||||
|
||||
# Declare ingestor ative
|
||||
# Declare ingestor active
|
||||
self.ingestor_manager.declare_active()
|
||||
|
||||
# Get slot lease
|
||||
@@ -160,12 +209,18 @@ class Ingestor:
|
||||
def manage_no_slots(self, number_of_slots: int):
|
||||
"""
|
||||
Manages the scenario where there are no slots assigned to the ingestor.
|
||||
This method checks if the ingestor is active (i.e., has no managed tags)
|
||||
and if the number of available slots is greater than zero. If both
|
||||
conditions are met, it attempts to acquire a slot lease and handles
|
||||
the acquired tags accordingly.
|
||||
|
||||
This method handles the case where an ingestor is active but has no
|
||||
allocated slots. It attempts to acquire a slot lease if slots are
|
||||
available in the system.
|
||||
|
||||
Args:
|
||||
number_of_slots (int): The number of available slots.
|
||||
number_of_slots (int): The number of available slots in the system.
|
||||
|
||||
Behavior:
|
||||
- Only attempts to acquire slots if the ingestor is currently active
|
||||
(has no managed tags) and there are slots available
|
||||
- Requests a single slot lease to begin processing
|
||||
"""
|
||||
|
||||
if not self.ingestor_manager.managed_tags and number_of_slots > 0:
|
||||
@@ -178,24 +233,30 @@ class Ingestor:
|
||||
self, available_slots: int, lacking_ingestors: int, slot_diff: int
|
||||
):
|
||||
"""
|
||||
Manages the allocation and deallocation of slot leases for ingestors based on
|
||||
the number of available slots, lacking ingestors, and slot differences.
|
||||
Manages the allocation and deallocation of slot leases for ingestors.
|
||||
|
||||
This method implements the load balancing logic for distributing OPC tag
|
||||
processing across multiple ingestor instances. It ensures optimal resource
|
||||
utilization and fair distribution of work.
|
||||
|
||||
Args:
|
||||
available_slots (int): The number of slots currently available for allocation.
|
||||
lacking_ingestors (int): The number of ingestors that are active and without slots.
|
||||
slot_diff (int): The difference between the total slots and the required slots.
|
||||
|
||||
Behavior:
|
||||
- If there are available slots and lacking ingestors, attempts to acquire slot leases
|
||||
for the available slots and processes the acquired tags.
|
||||
- If there are available slots and lacking ingestors, attempts to acquire
|
||||
slot leases for the available slots and processes the acquired tags.
|
||||
- If there are no lacking ingestors but there are extra slots (slot_diff > 0),
|
||||
releases the extra slot leases to ensure proper allocation.
|
||||
|
||||
Logs:
|
||||
- Logs the number of available slots when attempting to acquire leases.
|
||||
- Logs the number of extra slots when releasing leases.
|
||||
"""
|
||||
|
||||
if available_slots > 0 and lacking_ingestors > 0:
|
||||
# Some ingestors are innactive, so theres "available_slots" slots available
|
||||
# Some ingestors are inactive, so there are "available_slots" slots available
|
||||
self.logger.info(f"Slots available: {available_slots}")
|
||||
|
||||
# Get slot lease
|
||||
@@ -222,9 +283,23 @@ class Ingestor:
|
||||
|
||||
async def update_ingestor_manager(self, old_managed_tags: Dict[str, Any]):
|
||||
"""
|
||||
Updates the ingestor manager with the new managed tags.
|
||||
Updates the ingestor manager with new managed tags and handles configuration changes.
|
||||
|
||||
This method compares the current managed tags with the previous state and
|
||||
performs necessary operations to maintain synchronization:
|
||||
- Subscribes to newly allocated tags
|
||||
- Resubscribes to tags with changed configurations
|
||||
- Unsubscribes from deallocated tags
|
||||
|
||||
Args:
|
||||
old_managed_tags (Dict[str, Any]): The old managed tags.
|
||||
old_managed_tags (Dict[str, Any]): The previous state of managed tags.
|
||||
|
||||
Behavior:
|
||||
- Compares current and previous tag configurations
|
||||
- Establishes subscriptions for new tags
|
||||
- Updates subscriptions for modified tags
|
||||
- Removes subscriptions for deallocated tags
|
||||
- Updates metrics to reflect current state
|
||||
"""
|
||||
|
||||
self.logger.debug(
|
||||
@@ -267,18 +342,23 @@ class Ingestor:
|
||||
|
||||
async def loop(self):
|
||||
"""
|
||||
Executes the main loop for managing ingestors and slots.
|
||||
This method performs the following tasks:
|
||||
1. Declares the ingestor as active.
|
||||
2. Logs the start of the polling process for slot updates.
|
||||
3. Retrieves the list of active ingestors and the number of available slots.
|
||||
4. Handles scenarios where no slots are available.
|
||||
5. Calculates the difference between the number of slots and active ingestors,
|
||||
as well as the difference in managed tags.
|
||||
6. Manages leases based on the calculated differences.
|
||||
7. Logs the current state of active ingestors, slots, managed tags, and servers.
|
||||
8. Logs a message if no slots are acquired during the loop.
|
||||
9. Updates the configuration of OPC servers.
|
||||
Executes the main processing loop for managing ingestors and slots.
|
||||
|
||||
This method is the core of the ingestor's operation, performing the following
|
||||
tasks in each iteration:
|
||||
1. Declares the ingestor as active to maintain its presence in the system
|
||||
2. Polls for slot updates and manages resource allocation
|
||||
3. Handles scenarios where no slots are available
|
||||
4. Manages slot leases based on system load and available resources
|
||||
5. Updates OPC server configurations and checks server integrity
|
||||
6. Synchronizes managed tags with the current system state
|
||||
|
||||
The loop implements a sophisticated load balancing algorithm that:
|
||||
- Distributes OPC tag processing across multiple ingestor instances
|
||||
- Ensures optimal resource utilization
|
||||
- Maintains system stability during scaling operations
|
||||
- Provides real-time monitoring and metrics collection
|
||||
|
||||
This method is intended to be called repeatedly to ensure the ingestor
|
||||
manager operates correctly and maintains synchronization with the slots
|
||||
and OPC servers.
|
||||
|
||||
Reference in New Issue
Block a user