gitea_admin/sientia-dataops-opc-ingestor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1084

Browse Source 
Remove deprecated files and enhance documentation

- Deleted `coverage.sh`, `docker-compose.yaml`, `Dockerfile`, and simulator-related files to streamline the project structure.
- Updated `README.md` to provide a comprehensive overview of the OPC Ingestor, including features, architecture, installation, usage, and troubleshooting.
- Enhanced docstrings across various classes and methods in the `ingestor` module for better clarity and maintainability.
- Improved Prometheus metrics documentation in `metrics.py` to ensure proper monitoring and observability of the system.
This commit is contained in:
vitor-aignosi
2025-08-29 11:20:28 -03:00
parent f92ee20831
commit 8f6ba4ddcf
14 changed files with 1004 additions and 391 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
ingestor/managers/data_manager.py
View File

@@ -14,6 +14,38 @@ import os
class DataManager(BaseActivity):
"""
Manages data persistence and export operations for the OPC Ingestor.
The DataManager is responsible for:
- Storing OPC data in MongoDB for historical analysis and persistence
- Exporting data to Kafka for real-time streaming and downstream processing
- Managing database connections and ensuring data integrity
- Providing data access interfaces for other components
The manager supports both MongoDB and Kafka operations, with Kafka export
being optional and configurable. It implements retry logic for connection
failures and provides comprehensive error handling and notification.
Args:
kafka_servers (str): Comma-separated string of Kafka server addresses
mongo_connection_string (str): MongoDB connection string
mongo_database (str): MongoDB database name
export_to_kafka (bool): Whether to enable Kafka export functionality
metadata (dict): Application metadata for notifications and tracking
logger (Logger): Logger instance for application logging
notification_handler (NotificationHandler): Handler for sending notifications
Attributes:
pod_id (str): Pod identifier for metrics labeling
kafka_producer (KafkaProducer): Kafka producer instance for data export
export_to_kafka (bool): Whether Kafka export is enabled
connection_string (str): MongoDB connection string
database (str): MongoDB database name
mongo_client (MongoClient): MongoDB client instance
metadata (dict): Application metadata
"""
def __init__(
self,
kafka_servers: str,
@@ -25,15 +57,32 @@ class DataManager(BaseActivity):
notification_handler: NotificationHandler,
) -> None:
"""
Initializes the DataManager instance with a Kafka producer.
This constructor attempts to establish a connection to the specified Kafka servers
and initializes a Kafka producer for sending messages. It retries the connection
up to 3 times if the Kafka servers are unavailable.
Initializes the DataManager instance with Kafka and MongoDB connections.
This constructor attempts to establish connections to the specified services:
1. Kafka: Initializes producer with retry logic (up to 3 attempts)
2. MongoDB: Establishes connection and verifies server availability
The initialization process includes:
- Kafka producer setup with JSON serialization
- MongoDB client initialization and connection testing
- Metrics recording for connection status
- Error handling with notifications
Args:
kafka_servers (str): A comma-separated string of Kafka server addresses.
logger (Logger): A logger instance for logging messages.
kafka_servers (str): Comma-separated string of Kafka server addresses
mongo_connection_string (str): MongoDB connection string
mongo_database (str): MongoDB database name
export_to_kafka (bool): Whether to enable Kafka export
metadata (dict): Application metadata
logger (Logger): Logger instance
notification_handler (NotificationHandler): Notification handler
Raises:
NoBrokersAvailable: If the connection to Kafka servers fails after 3 attempts.
Metrics:
- KAFKA_CONNECTION_STATUS: Set to 1 on successful connection, 0 on failure
"""
self.pod_id = os.getenv("HOSTNAME", "localhost")

56
ingestor/managers/ingestor_manager.py
View File

@@ -13,6 +13,50 @@ import ingestor.metrics as metrics
class IngestorManager(BaseActivity):
"""
Central coordinator for managing OPC data ingestion operations.
The IngestorManager orchestrates the interaction between different components:
- DataManager: Handles data persistence and Kafka export
- OPC Managers: Manage individual OPC UA server connections
- ResourceManager: Coordinates slot leasing and load balancing
This class implements a slot-based architecture where:
- Each slot represents a collection of OPC tags from one or more servers
- Slots are distributed across multiple ingestor instances for load balancing
- Dynamic slot allocation ensures optimal resource utilization
Key Responsibilities:
- Slot lease management and distribution
- OPC server connection lifecycle management
- Tag subscription coordination
- System health monitoring and integrity checks
- Load balancing across multiple ingestor instances
Args:
kafka_servers (str): Comma-separated list of Kafka server addresses
redis_data (dict): Redis connection parameters (host, port, username, password)
lease_ttl (int): Time-to-live for slot leases in seconds
heartbeat_ttl (int): Time-to-live for heartbeat signals in seconds
poll_interval (int): Main loop polling interval in seconds
mongo_connection_string (str): MongoDB connection string
mongo_database (str): MongoDB database name
metadata (dict): Application metadata for notifications and tracking
logger (Logger): Logger instance for application logging
notification_handler (NotificationHandler): Handler for sending notifications
export_to_kafka (bool): Whether to export data to Kafka
Attributes:
data_manager (DataManager): Manages data persistence and Kafka export
opc_managers (dict): Dictionary of OPC managers keyed by server name
resource_manager (ResourceManager): Manages Redis-based resource coordination
number_of_slots (int): Total number of slots configured in the system
poll_interval (int): Main loop polling interval
managed_tags (dict): Currently managed tags organized by slot
opc_servers (dict): OPC server configurations
metadata (dict): Application metadata
"""
def __init__(self,
kafka_servers: str, redis_data: dict,
lease_ttl: int, heartbeat_ttl: int,
@@ -61,6 +105,10 @@ class IngestorManager(BaseActivity):
async def initialize_opc_from_config(self, server_config: dict) -> OpcManager | None:
"""
Initializes an OPC Manager instance using the provided server configuration.
This method creates and configures an OPC Manager for a specific OPC UA server,
establishing the connection and preparing it for tag subscriptions.
Args:
server_config (dict): A dictionary containing the OPC server configuration.
Expected keys include:
@@ -70,11 +118,15 @@ class IngestorManager(BaseActivity):
- 'cert_path' (str, optional): Path to the client certificate file.
- 'private_key_path' (str, optional): Path to the private key file.
- 'server_cert_path' (str, optional): Path to the server certificate file.
data_manager (DataManager): An instance of the DataManager to handle data operations.
logger (Logger): A logger instance for logging messages.
Returns:
OpcManager | None: An initialized OpcManager instance if successful,
otherwise None if an error occurs during initialization.
Raises:
Exception: If OPC manager initialization fails, the error is logged and
a notification is sent, but the method returns None to allow
the system to continue operating with other servers.
"""
try:

88
ingestor/managers/opc_manager.py
View File

@@ -13,6 +13,47 @@ import ingestor.metrics as metrics
class OpcManager(BaseActivity):
"""
Manages OPC UA server connections and tag subscriptions.
The OpcManager is responsible for:
- Establishing and maintaining secure connections to OPC UA servers
- Managing tag subscriptions and data collection
- Handling server reconnection and error recovery
- Processing OPC data and forwarding it to the data manager
- Monitoring connection health and performance metrics
The manager supports both secure and unsecured connections, with optional
certificate-based authentication for enhanced security.
Args:
name (str): Unique identifier for the OPC server
url (str): OPC UA server endpoint URL
data_manager (DataManager): Manager for data persistence and export
logger (Logger): Logger instance for application logging
server_uri (str): OPC UA server application URI
notification_handler (NotificationHandler): Handler for sending notifications
metadata (dict): Application metadata for notifications and tracking
cert_path (str, optional): Path to client certificate file for secure connections
private_key_path (str, optional): Path to client private key file
server_cert_path (str, optional): Path to server certificate file for validation
Attributes:
url (str): OPC UA server endpoint URL
name (str): Unique identifier for the OPC server
server_uri (str): OPC UA server application URI
data_queue (dict): Queue for buffering OPC data before processing
non_receive_count (int): Counter for cycles without data reception
client (Client): OPC UA client instance
cert_path (str): Path to client certificate file
private_key_path (str): Path to client private key file
server_cert_path (str): Path to server certificate file
nodes (dict): Dictionary of OPC node references
subscriptions (dict): Active OPC subscriptions
data_manager (DataManager): Manager for data persistence and export
metadata (dict): Application metadata
"""
def __init__(self, name: str, url: str, data_manager: DataManager, logger: Logger,
server_uri: str, notification_handler: NotificationHandler, metadata: dict,
cert_path: str = None, private_key_path: str = None, server_cert_path: str = None):
@@ -40,11 +81,27 @@ class OpcManager(BaseActivity):
pod_id=self.pod_id, server_name=self.name).set(0)
def __str__(self):
"""
String representation of the OPC Manager.
Returns:
str: Human-readable representation showing server details and current state.
"""
return f"OpcManager(name={self.name}, url={self.url}, server_uri={self.server_uri})\n" \
f"nodes={self.nodes}, subscriptions={self.subscriptions}"
async def shutdown(self):
"""Comprehensive cleanup method"""
"""
Comprehensive cleanup method for graceful shutdown.
This method ensures proper cleanup of all OPC UA resources:
- Closes active subscriptions
- Disconnects from the OPC server
- Releases allocated resources
Should be called before the application terminates to prevent resource leaks
and ensure clean disconnection from OPC servers.
"""
try:
await self.disconnect()
@@ -54,21 +111,24 @@ class OpcManager(BaseActivity):
async def set_security(self):
"""
Configures the security settings for the OPC UA client.
This method sets up the security policy, certificates, and timeouts
required for establishing a secure connection with the OPC UA server.
It implements Basic256 security policy with certificate-based authentication.
Raises:
ValueError: If either the certificate path or private key path is not provided.
Attributes:
cert_path (str): Path to the client's certificate file.
private_key_path (str): Path to the client's private key file.
server_cert_path (str, optional): Path to the server's certificate file.
server_uri (str): The URI of the server to be used as the application URI.
client (opcua.Client): The OPC UA client instance.
logger (logging.Logger): Logger instance for logging information.
Security Settings:
- Security Policy: Basic256
- Secure Channel Timeout: 10,000,000 ms
- Session Timeout: 10,000,000 ms
The method configures:
- Client application URI
- Certificate-based authentication
- Server certificate validation (if provided)
- Connection timeouts for stability
"""
if not all([self.cert_path, self.private_key_path]):
@@ -93,11 +153,23 @@ class OpcManager(BaseActivity):
async def connect(self):
"""
Establishes a connection to the OPC server.
This method initializes the OPC client using the provided URL and
sets up security if a certificate path is specified. It then
attempts to connect to the server and logs the connection status.
The connection process includes:
1. Client initialization with server URL
2. Security configuration (if certificates are provided)
3. Connection establishment
4. Metrics recording for monitoring
Raises:
Exception: If the connection to the OPC server fails.
Metrics:
- OPC_CONNECTIONS_TOTAL: Incremented on connection attempt
- OPC_CONNECTION_STATUS: Set to 1 on successful connection
"""
metrics.OPC_CONNECTIONS_TOTAL.labels(

238
ingestor/managers/resource_manager.py
View File

@@ -10,6 +10,39 @@ from sientia_do.temporal.activities.base import BaseActivity
class ResourceManager(BaseActivity):
"""
Manages Redis-based resource coordination and slot leasing for the OPC Ingestor.
The ResourceManager is responsible for:
- Coordinating slot allocation across multiple ingestor instances
- Managing lease lifecycles and heartbeats for load balancing
- Providing distributed locking and resource management
- Monitoring Redis operations and connection health
The manager implements a sophisticated slot leasing system that enables:
- Dynamic load distribution across multiple ingestor instances
- Automatic failover and recovery from instance failures
- Fair resource allocation based on system capacity
- Real-time monitoring of system health and performance
Args:
host (str): Redis server hostname
port (int): Redis server port
lease_ttl (int): Time-to-live for slot leases in seconds
heartbeat_ttl (int): Time-to-live for heartbeat signals in seconds
metadata (dict): Application metadata for notifications and tracking
logger (Logger): Logger instance for application logging
notification_handler (NotificationHandler): Handler for sending notifications
username (str, optional): Redis username for authentication
password (str, optional): Redis password for authentication
Attributes:
redis (Redis): Redis client instance
lease_ttl (int): Time-to-live for slot leases
heartbeat_ttl (int): Time-to-live for heartbeat signals
metadata (dict): Application metadata
"""
def __init__(
self,
host: str,
@@ -22,6 +55,31 @@ class ResourceManager(BaseActivity):
username: str | None = None,
password: str | None = None,
) -> None:
"""
Initializes the ResourceManager with Redis connection and configuration.
This constructor establishes a connection to Redis and verifies connectivity
by performing a ping operation. It sets up the connection with optional
authentication and records the connection status in metrics.
Args:
host (str): Redis server hostname
port (int): Redis server port
lease_ttl (int): Time-to-live for slot leases in seconds
heartbeat_ttl (int): Time-to-live for heartbeat signals in seconds
metadata (dict): Application metadata
logger (Logger): Logger instance
notification_handler (NotificationHandler): Notification handler
username (str, optional): Redis username for authentication
password (str, optional): Redis password for authentication
Raises:
Exception: If Redis connection fails, the error is logged and metrics
are updated before re-raising the exception.
Metrics:
- REDIS_CONNECTION_STATUS: Set to 1 on successful connection, 0 on failure
"""
BaseActivity.__init__(self, logger=logger,
notification_handler=notification_handler,
set_error_counter=True)
@@ -45,7 +103,32 @@ class ResourceManager(BaseActivity):
self.metadata = metadata
def _execute_redis_op(self, operation_name: str, func, *args, **kwargs):
"""Wrapper to execute Redis operations and record metrics."""
"""
Wrapper to execute Redis operations and record metrics.
This method provides a unified interface for Redis operations that:
- Records operation timing and success/failure metrics
- Handles error notifications consistently
- Ensures all Redis operations are properly monitored
Args:
operation_name (str): Name of the Redis operation for metrics labeling
func: The Redis function to execute
*args: Positional arguments for the Redis function
**kwargs: Keyword arguments for the Redis function
Returns:
The result of the Redis operation
Raises:
Exception: Re-raises any exception from the Redis operation after
recording error metrics and sending notifications.
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented on successful operations
- REDIS_OPERATIONS_DURATION: Records operation timing
- REDIS_OPERATIONS_ERRORS: Incremented on operation failures
"""
start_time = time()
try:
result = func(*args, **kwargs)
@@ -73,11 +156,20 @@ class ResourceManager(BaseActivity):
def get(self, key: str) -> dict:
"""
Retrieve a value from Redis by its key and return it as a dictionary.
This method fetches a value from Redis and attempts to parse it as JSON.
If the key doesn't exist or the value is empty, it returns None.
Args:
key (str): The key to look up in Redis.
Returns:
dict: The value associated with the key, parsed as a dictionary,
or None if the key does not exist or the value is empty.
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="get"
- REDIS_OPERATIONS_DURATION: Records timing for get operations
"""
history = self._execute_redis_op("get", self.redis.get, key)
@@ -86,10 +178,19 @@ class ResourceManager(BaseActivity):
def get_tag_slot(self, id: str) -> dict:
"""
Retrieve the tag slot information for a given ID.
This method constructs the Redis key for a tag slot and retrieves
the associated configuration information.
Args:
id (str): The unique identifier of the tag slot to retrieve.
Returns:
dict: A dictionary containing the tag slot information associated with the given ID.
dict: A dictionary containing the tag slot information associated
with the given ID, or None if not found.
The method constructs the key using the pattern "slot:opc_tags:{id}"
and delegates to the get() method for the actual Redis operation.
"""
return self.get(f"slot:opc_tags:{id}")
@@ -97,12 +198,20 @@ class ResourceManager(BaseActivity):
def ingestor_heartbeat(self) -> None:
"""
Sends a heartbeat signal to Redis to indicate that the ingestor is active.
This method sets a key in Redis with a specific format that includes the
ingestor's pod ID. The key is set with a value of 1 and an expiration
time defined by `self.heartbeat_ttl`. This allows monitoring systems to
track the activity and health of the ingestor.
Returns:
None
The heartbeat mechanism enables:
- Load balancers to identify active ingestor instances
- Health monitoring systems to detect failed instances
- Automatic failover and recovery mechanisms
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="set"
- REDIS_OPERATIONS_DURATION: Records timing for heartbeat operations
"""
self._execute_redis_op(
@@ -115,14 +224,28 @@ class ResourceManager(BaseActivity):
def lease_tag(self, tag_id: str) -> bool:
"""
Attempts to lease a tag by setting a key in Redis with a specified TTL (time-to-live).
This method uses the Redis `SET` command with the `NX` option to ensure that the key
is only set if it does not already exist. The key is set with an expiration time
defined by `lease_ttl`.
Attempts to lease a tag by setting a key in Redis with a specified TTL.
This method uses the Redis `SET` command with the `NX` option to ensure that
the key is only set if it does not already exist. The key is set with an
expiration time defined by `lease_ttl`. This implements a distributed
locking mechanism for tag allocation.
Args:
tag_id (str): The unique identifier of the tag to be leased.
Returns:
bool: True if the lease was successfully acquired, False otherwise.
bool: True if the lease was successfully acquired, False if the tag
is already leased by another ingestor.
The leasing mechanism ensures:
- Only one ingestor can process a specific tag at a time
- Automatic lease expiration prevents deadlocks
- Fair distribution of tags across available ingestor instances
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="set_nx"
- REDIS_OPERATIONS_DURATION: Records timing for lease operations
"""
return self._execute_redis_op(
@@ -137,13 +260,26 @@ class ResourceManager(BaseActivity):
def renew_tag_lease(self, tag_id: str) -> bool:
"""
Renews the lease for a specific OPC tag if the current pod holds the lease.
This method checks if the current pod (identified by `self.pod_id`) holds the lease
for the given OPC tag. If so, it extends the lease by resetting its expiration time
in Redis to the configured lease TTL (`self.lease_ttl`).
This method checks if the current pod (identified by `self.pod_id`) holds
the lease for the given OPC tag. If so, it extends the lease by resetting
its expiration time in Redis to the configured lease TTL.
Args:
tag_id (str): The identifier of the OPC tag whose lease is to be renewed.
Returns:
bool: True if the lease was successfully renewed, False otherwise.
bool: True if the lease was successfully renewed, False if the current
pod doesn't hold the lease or renewal failed.
Lease renewal is essential for:
- Maintaining continuous tag processing without interruptions
- Preventing lease expiration during long-running operations
- Ensuring system stability and reliability
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="get" and "expire"
- REDIS_OPERATIONS_DURATION: Records timing for renewal operations
"""
current = self._execute_redis_op(
@@ -159,11 +295,22 @@ class ResourceManager(BaseActivity):
def drop_tag_lease(self, tag_id: str) -> None:
"""
Drops the lease for a specific OPC tag.
This method removes the lease for the given OPC tag by deleting the corresponding key in Redis.
This method removes the lease for the given OPC tag by deleting the
corresponding key in Redis. This is typically called when an ingestor
is shutting down or when it needs to release a tag for reallocation.
Args:
tag_id (str): The identifier of the OPC tag whose lease is to be dropped.
Returns:
None
Lease dropping enables:
- Graceful shutdown of ingestor instances
- Dynamic reallocation of tags for load balancing
- Recovery from failed or unresponsive ingestor instances
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="delete"
- REDIS_OPERATIONS_DURATION: Records timing for lease dropping operations
"""
self._execute_redis_op("delete", self.redis.delete,
@@ -172,21 +319,47 @@ class ResourceManager(BaseActivity):
def get_all_ingestors(self) -> List[str]:
"""
Retrieves all active ingestors from Redis.
This method fetches all keys in Redis that match the pattern for ingestor leases
and returns a list of active ingestors.
This method fetches all keys in Redis that match the pattern for ingestor
heartbeats and returns a list of active ingestor identifiers. The method
uses the pattern "heartbeat:ingestor:*" to find all active instances.
Returns:
list: A list of active ingestors.
List[str]: A list of active ingestor identifiers, extracted from
the Redis keys by removing the "heartbeat:ingestor:" prefix.
This information is used for:
- Load balancing calculations
- System health monitoring
- Resource allocation decisions
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="keys"
- REDIS_OPERATIONS_DURATION: Records timing for ingestor discovery
"""
return self._execute_redis_op("keys", self.redis.keys, "heartbeat:ingestor:*")
def get_all_slots(self) -> List[str]:
"""
Retrieves the number of slots available in Redis.
This method counts the number of keys in Redis that match the pattern for OPC tag leases
and returns the count.
Retrieves all available slots from Redis.
This method fetches all keys in Redis that match the pattern for OPC tag
slots and returns a list of slot identifiers. The method uses the pattern
"slot:opc_tags:*" to find all configured slots.
Returns:
int: The number of slots available.
List[str]: A list of slot identifiers, extracted from the Redis keys
by removing the "slot:opc_tags:" prefix.
Slot information is used for:
- Resource allocation planning
- Load balancing across ingestor instances
- System capacity monitoring
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="keys"
- REDIS_OPERATIONS_DURATION: Records timing for slot discovery
"""
return self._execute_redis_op("keys", self.redis.keys, "slot:opc_tags:*")
@@ -194,10 +367,23 @@ class ResourceManager(BaseActivity):
def get_all_leases(self) -> List[str]:
"""
Retrieves all active leases from Redis.
This method fetches all keys in Redis that match the pattern for OPC tag leases
and returns a list of active leases.
This method fetches all keys in Redis that match the pattern for OPC tag
leases and returns a list of lease identifiers. The method uses the pattern
"lease:opc_tags:*" to find all active leases.
Returns:
list: A list of active leases.
List[str]: A list of lease identifiers, extracted from the Redis keys
by removing the "lease:opc_tags:" prefix.
Lease information is used for:
- Current resource utilization monitoring
- Load balancing calculations
- System health and performance analysis
Metrics:
- REDIS_OPERATIONS_TOTAL: Incremented with operation="keys"
- REDIS_OPERATIONS_DURATION: Records timing for lease discovery
"""
return self._execute_redis_op("keys", self.redis.keys, "lease:opc_tags:*")