SIENTIAPDE-1084

Remove deprecated files and enhance documentation

- Deleted `coverage.sh`, `docker-compose.yaml`, `Dockerfile`, and simulator-related files to streamline the project structure.
- Updated `README.md` to provide a comprehensive overview of the OPC Ingestor, including features, architecture, installation, usage, and troubleshooting.
- Enhanced docstrings across various classes and methods in the `ingestor` module for better clarity and maintainability.
- Improved Prometheus metrics documentation in `metrics.py` to ensure proper monitoring and observability of the system.
This commit is contained in:
vitor-aignosi
2025-08-29 11:20:28 -03:00
parent f92ee20831
commit 8f6ba4ddcf
14 changed files with 1004 additions and 391 deletions

View File

@@ -13,6 +13,50 @@ import ingestor.metrics as metrics
class IngestorManager(BaseActivity):
"""
Central coordinator for managing OPC data ingestion operations.
The IngestorManager orchestrates the interaction between different components:
- DataManager: Handles data persistence and Kafka export
- OPC Managers: Manage individual OPC UA server connections
- ResourceManager: Coordinates slot leasing and load balancing
This class implements a slot-based architecture where:
- Each slot represents a collection of OPC tags from one or more servers
- Slots are distributed across multiple ingestor instances for load balancing
- Dynamic slot allocation ensures optimal resource utilization
Key Responsibilities:
- Slot lease management and distribution
- OPC server connection lifecycle management
- Tag subscription coordination
- System health monitoring and integrity checks
- Load balancing across multiple ingestor instances
Args:
kafka_servers (str): Comma-separated list of Kafka server addresses
redis_data (dict): Redis connection parameters (host, port, username, password)
lease_ttl (int): Time-to-live for slot leases in seconds
heartbeat_ttl (int): Time-to-live for heartbeat signals in seconds
poll_interval (int): Main loop polling interval in seconds
mongo_connection_string (str): MongoDB connection string
mongo_database (str): MongoDB database name
metadata (dict): Application metadata for notifications and tracking
logger (Logger): Logger instance for application logging
notification_handler (NotificationHandler): Handler for sending notifications
export_to_kafka (bool): Whether to export data to Kafka
Attributes:
data_manager (DataManager): Manages data persistence and Kafka export
opc_managers (dict): Dictionary of OPC managers keyed by server name
resource_manager (ResourceManager): Manages Redis-based resource coordination
number_of_slots (int): Total number of slots configured in the system
poll_interval (int): Main loop polling interval
managed_tags (dict): Currently managed tags organized by slot
opc_servers (dict): OPC server configurations
metadata (dict): Application metadata
"""
def __init__(self,
kafka_servers: str, redis_data: dict,
lease_ttl: int, heartbeat_ttl: int,
@@ -61,6 +105,10 @@ class IngestorManager(BaseActivity):
async def initialize_opc_from_config(self, server_config: dict) -> OpcManager | None:
"""
Initializes an OPC Manager instance using the provided server configuration.
This method creates and configures an OPC Manager for a specific OPC UA server,
establishing the connection and preparing it for tag subscriptions.
Args:
server_config (dict): A dictionary containing the OPC server configuration.
Expected keys include:
@@ -70,11 +118,15 @@ class IngestorManager(BaseActivity):
- 'cert_path' (str, optional): Path to the client certificate file.
- 'private_key_path' (str, optional): Path to the private key file.
- 'server_cert_path' (str, optional): Path to the server certificate file.
data_manager (DataManager): An instance of the DataManager to handle data operations.
logger (Logger): A logger instance for logging messages.
Returns:
OpcManager | None: An initialized OpcManager instance if successful,
otherwise None if an error occurs during initialization.
Raises:
Exception: If OPC manager initialization fails, the error is logged and
a notification is sent, but the method returns None to allow
the system to continue operating with other servers.
"""
try: