SIENTIAPDE-1974: fix(slots): guard every reader unsubscribe and count failures
Second code review round (xhigh) of057c1fd+085e609: 5 findings. Fixed: - manage_server's subscribe-failure path called reader.unsubscribe() directly; with the legacy OpcManager a dead session made it raise out of the except block and stop the main loop. Both paths now go through _unsubscribe_reader, the single guarded per-reader unsubscribe. - An unsubscribe failure also increments OPC_SUBSCRIPTION_ERRORS (same tags as the subscription failures), besides the OPC_UNSUBSCRIBE_ERROR notification. - The contract test creates the subscriptions through the readers' real create_subscription (fake pool / client) instead of writing engine internals, and checks the pool pin/unpin pair. Documented (no code change): - If delete() fails while the session is still alive (e.g. a timeout), the server-side subscription may keep delivering data until that session ends (reconnection or pod restart); noted in _unsubscribe_reader. - manage_leases now unsubscribes before releasing the leases: if Redis fails while releasing them, the slots stay unread until LEASE_TTL expires, which is preferred to two pods reading the same slot. 284 unit tests; the 2 new behavior tests fail on085e609. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -549,10 +549,8 @@ class IngestorManager(SientiaMonitoring):
|
||||
also left the subscription open on the OPC server. Unsubscribing a slot that no
|
||||
reader holds is a no-op.
|
||||
|
||||
A reader that fails to unsubscribe (e.g. the legacy OpcManager, whose delete()
|
||||
raises when the server session is gone) is notified and logged, and the slot is
|
||||
dropped from its books, as the v2 engine already does, so it can be subscribed
|
||||
again. The other readers are still unsubscribed and the main loop goes on.
|
||||
A reader that fails to unsubscribe is handled by _unsubscribe_reader: the other
|
||||
readers are still unsubscribed and the main loop goes on.
|
||||
|
||||
Args:
|
||||
slot (str): The name of the slot to unsubscribe.
|
||||
@@ -560,28 +558,56 @@ class IngestorManager(SientiaMonitoring):
|
||||
Side Effects:
|
||||
- Removes the slot's subscription from every OPC reader that has one
|
||||
- Cleans up subscription resources on the OPC servers
|
||||
- Sends OPC_UNSUBSCRIBE_ERROR_<slot>:<server> when a reader fails
|
||||
"""
|
||||
|
||||
# list() snapshots the readers: the loop awaits, so a reader added or removed
|
||||
# meanwhile must not break the iteration.
|
||||
for server, reader in list(self.opc_managers.items()): # NOSONAR
|
||||
if slot not in reader.subscriptions:
|
||||
continue
|
||||
try:
|
||||
await reader.unsubscribe(slot)
|
||||
except Exception as e:
|
||||
trace = traceback.format_exc()
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_UNSUBSCRIBE_ERROR_{slot}:{server}',
|
||||
message=f'Failed to unsubscribe slot {slot} from {server}: {e}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=trace,
|
||||
)
|
||||
self.logger.error(trace)
|
||||
reader.subscriptions.pop(slot, None)
|
||||
if slot in reader.subscriptions:
|
||||
await self._unsubscribe_reader(server, reader, slot)
|
||||
|
||||
async def _unsubscribe_reader(
|
||||
self, server: str, reader: ServerReader | OpcManager, slot: str
|
||||
) -> None:
|
||||
"""
|
||||
Unsubscribe one reader from a slot without letting a failure stop the main loop.
|
||||
|
||||
A failure (e.g. the legacy OpcManager, whose delete() raises when the server
|
||||
session is gone) increments OPC_SUBSCRIPTION_ERRORS, sends
|
||||
OPC_UNSUBSCRIBE_ERROR_<slot>:<server> and is logged. The slot is dropped from the
|
||||
reader's books, as the v2 engine already does, so it can be subscribed again.
|
||||
|
||||
Known limitation: if delete() failed while the session is still alive (e.g. a
|
||||
timeout), the server-side subscription may keep delivering data until that
|
||||
session ends (reconnection or pod restart).
|
||||
|
||||
Args:
|
||||
server (str): The reader's key in opc_managers
|
||||
reader (ServerReader | OpcManager): The OPC reader
|
||||
slot (str): The slot to unsubscribe
|
||||
"""
|
||||
try:
|
||||
await reader.unsubscribe(slot)
|
||||
except Exception as e:
|
||||
await self.emit_metric(
|
||||
metric_object=metrics.OPC_SUBSCRIPTION_ERRORS,
|
||||
tags={
|
||||
'pod_id': self.pod_id,
|
||||
'server': server,
|
||||
'slot': slot,
|
||||
},
|
||||
)
|
||||
trace = traceback.format_exc()
|
||||
await self.send_notification_async(
|
||||
metadata=self.metadata,
|
||||
notification_id=f'OPC_UNSUBSCRIBE_ERROR_{slot}:{server}',
|
||||
message=f'Failed to unsubscribe slot {slot} from {server}: {e}',
|
||||
block='opc_manager',
|
||||
level=NotificationLevel.ERROR,
|
||||
attachment_content=trace,
|
||||
)
|
||||
self.logger.error(trace)
|
||||
reader.subscriptions.pop(slot, None)
|
||||
|
||||
async def _unsubscribe_removed_slots(self, slots: list[str]) -> None:
|
||||
"""Unsubscribe and forget slots whose configuration is gone from Redis."""
|
||||
@@ -796,7 +822,7 @@ class IngestorManager(SientiaMonitoring):
|
||||
self.logger.error(trace)
|
||||
|
||||
self.logger.warning(f'Removing subscription from server {server} for slot {slot}')
|
||||
await self.opc_managers[server].unsubscribe(slot)
|
||||
await self._unsubscribe_reader(server, self.opc_managers[server], slot)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
Reference in New Issue
Block a user