50578fdba8
Swaps the legacy OpcManager read path for the gateway's SubscriptionEngine over the shared OpcConnectionPool. The OPC client certificate now always comes from MinIO + Mongo (CertProvider, Basic256Sha256) — no more certs/ shipped in the Gitea repository, so a new version branch no longer breaks the reads. - io/slots.py: parse_slot accepts slot v2 (names[]) and converts the legacy v1 (tag_name + topics; cert_path => use_security, path dropped), so gateway and Orchestrator can be deployed in any order. - managers/server_reader.py: new per-server reader with the OpcManager interface the IngestorManager already drives (connect, create_subscription, subscribe, unsubscribe, check_cycles, check_opc_listenning, shutdown). - IngestorManager owns the read path: lazy-MinIO CertProvider, the pool and a read DataSink, built without touching the network. Slots from Redis are normalized on lease and refresh; a malformed slot is notified and the last good config is kept. - GatewayRuntime borrows the pool (writes/gRPC) instead of building it, and no longer closes it on stop: a Postgres/NATS failure can't stop the reads. - pool: pin/unpin per active subscription so release_idle never evicts a subscribed connection; public evict() for lost servers / changed config. - SubscriptionEngine pins on subscribe, unpins on unsubscribe, and drops a subscription locally even when the server-side delete fails. - DataSink: model_id only written when the name has one, keeping v1 documents identical to the legacy ones. - config: CERT_BUCKET default is the legacy streamlit-connectors bucket (D7). - Removes OpcManager, its tests and scripts/generate-and-push-opc-cert.sh; README/CLAUDE.md document the MinIO certificate and the slot v2 format. Tests: 218 unit tests pass; ruff check/format clean. Local end-to-end against an asyncua server: v1 slot with two topics -> one doc per collection, legacy doc shape, idle sweep keeps the subscription, lost-server shutdown and re-initialization work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
46 lines
1.8 KiB
Python
46 lines
1.8 KiB
Python
"""Parse the slot written by the Orchestrator to Redis into the slot v2 structure (contract C1).
|
|
|
|
Both shapes are accepted so the gateway and the Orchestrator can be deployed in
|
|
any order (SIENTIAPDE-1974):
|
|
|
|
- v2: each tag carries `names` (one entry per collection / tag name / model);
|
|
- v1 (legacy): each tag carries `tag_name` + `topics`, and the server carries the
|
|
file-based `cert_path`. v1 is converted to v2: one `names` entry per topic, and
|
|
`use_security` is true when the server had a certificate path (the certificate
|
|
itself now always comes from MinIO via CertProvider, never from that path).
|
|
"""
|
|
|
|
SERVER_FIELDS = ('server_id', 'url', 'server_uri', 'use_security', 'subscription_period_ms')
|
|
|
|
|
|
def _parse_names(address: str, tag: dict) -> list[dict]:
|
|
names = tag.get('names')
|
|
if names:
|
|
return names
|
|
topics = tag.get('topics')
|
|
if topics and tag.get('tag_name'):
|
|
return [{'collection': topic, 'tag_name': tag['tag_name']} for topic in topics]
|
|
raise ValueError(f'tag {address!r} has no names')
|
|
|
|
|
|
def _parse_tag(address: str, tag: dict) -> dict:
|
|
return {
|
|
'frequency': tag['frequency'],
|
|
'names': _parse_names(address, tag),
|
|
'active_read': tag.get('active_read', {'enabled': False}),
|
|
}
|
|
|
|
|
|
def _parse_server(name: str, server: dict) -> dict:
|
|
parsed = {field: server[field] for field in SERVER_FIELDS if field in server}
|
|
parsed['name'] = server.get('name', name)
|
|
if 'use_security' not in parsed:
|
|
parsed['use_security'] = bool(server.get('cert_path'))
|
|
parsed['tags'] = {addr: _parse_tag(addr, tag) for addr, tag in server['tags'].items()}
|
|
return parsed
|
|
|
|
|
|
def parse_slot(raw: dict) -> dict:
|
|
"""Normalize a raw slot (v1 or v2) into `{server_name: {..., tags: {addr: tag}}}`."""
|
|
return {name: _parse_server(name, server) for name, server in raw.items()}
|