SIENTIAPDE-1974: fix(slots): guard every reader unsubscribe and count failures

Second code review round (xhigh) of 057c1fd + 085e609: 5 findings.

Fixed:
- manage_server's subscribe-failure path called reader.unsubscribe() directly;
  with the legacy OpcManager a dead session made it raise out of the except
  block and stop the main loop. Both paths now go through _unsubscribe_reader,
  the single guarded per-reader unsubscribe.
- An unsubscribe failure also increments OPC_SUBSCRIPTION_ERRORS (same tags as
  the subscription failures), besides the OPC_UNSUBSCRIBE_ERROR notification.
- The contract test creates the subscriptions through the readers' real
  create_subscription (fake pool / client) instead of writing engine
  internals, and checks the pool pin/unpin pair.

Documented (no code change):
- If delete() fails while the session is still alive (e.g. a timeout), the
  server-side subscription may keep delivering data until that session ends
  (reconnection or pod restart); noted in _unsubscribe_reader.
- manage_leases now unsubscribes before releasing the leases: if Redis fails
  while releasing them, the slots stay unread until LEASE_TTL expires, which is
  preferred to two pods reading the same slot.

284 unit tests; the 2 new behavior tests fail on 085e609.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Eduardo Rios
2026-10-07 16:15:06 -03:00
parent 085e6090c8
commit 4235ce8edb
3 changed files with 100 additions and 30 deletions

View File

@@ -549,10 +549,8 @@ class IngestorManager(SientiaMonitoring):
also left the subscription open on the OPC server. Unsubscribing a slot that no
reader holds is a no-op.
A reader that fails to unsubscribe (e.g. the legacy OpcManager, whose delete()
raises when the server session is gone) is notified and logged, and the slot is
dropped from its books, as the v2 engine already does, so it can be subscribed
again. The other readers are still unsubscribed and the main loop goes on.
A reader that fails to unsubscribe is handled by _unsubscribe_reader: the other
readers are still unsubscribed and the main loop goes on.
Args:
slot (str): The name of the slot to unsubscribe.
@@ -560,28 +558,56 @@ class IngestorManager(SientiaMonitoring):
Side Effects:
- Removes the slot's subscription from every OPC reader that has one
- Cleans up subscription resources on the OPC servers
- Sends OPC_UNSUBSCRIBE_ERROR_<slot>:<server> when a reader fails
"""
# list() snapshots the readers: the loop awaits, so a reader added or removed
# meanwhile must not break the iteration.
for server, reader in list(self.opc_managers.items()): # NOSONAR
if slot not in reader.subscriptions:
continue
try:
await reader.unsubscribe(slot)
except Exception as e:
trace = traceback.format_exc()
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_UNSUBSCRIBE_ERROR_{slot}:{server}',
message=f'Failed to unsubscribe slot {slot} from {server}: {e}',
block='opc_manager',
level=NotificationLevel.ERROR,
attachment_content=trace,
)
self.logger.error(trace)
reader.subscriptions.pop(slot, None)
if slot in reader.subscriptions:
await self._unsubscribe_reader(server, reader, slot)
async def _unsubscribe_reader(
self, server: str, reader: ServerReader | OpcManager, slot: str
) -> None:
"""
Unsubscribe one reader from a slot without letting a failure stop the main loop.
A failure (e.g. the legacy OpcManager, whose delete() raises when the server
session is gone) increments OPC_SUBSCRIPTION_ERRORS, sends
OPC_UNSUBSCRIBE_ERROR_<slot>:<server> and is logged. The slot is dropped from the
reader's books, as the v2 engine already does, so it can be subscribed again.
Known limitation: if delete() failed while the session is still alive (e.g. a
timeout), the server-side subscription may keep delivering data until that
session ends (reconnection or pod restart).
Args:
server (str): The reader's key in opc_managers
reader (ServerReader | OpcManager): The OPC reader
slot (str): The slot to unsubscribe
"""
try:
await reader.unsubscribe(slot)
except Exception as e:
await self.emit_metric(
metric_object=metrics.OPC_SUBSCRIPTION_ERRORS,
tags={
'pod_id': self.pod_id,
'server': server,
'slot': slot,
},
)
trace = traceback.format_exc()
await self.send_notification_async(
metadata=self.metadata,
notification_id=f'OPC_UNSUBSCRIBE_ERROR_{slot}:{server}',
message=f'Failed to unsubscribe slot {slot} from {server}: {e}',
block='opc_manager',
level=NotificationLevel.ERROR,
attachment_content=trace,
)
self.logger.error(trace)
reader.subscriptions.pop(slot, None)
async def _unsubscribe_removed_slots(self, slots: list[str]) -> None:
"""Unsubscribe and forget slots whose configuration is gone from Redis."""
@@ -796,7 +822,7 @@ class IngestorManager(SientiaMonitoring):
self.logger.error(trace)
self.logger.warning(f'Removing subscription from server {server} for slot {slot}')
await self.opc_managers[server].unsubscribe(slot)
await self._unsubscribe_reader(server, self.opc_managers[server], slot)
return 2
return 0