50578fdba8
Swaps the legacy OpcManager read path for the gateway's SubscriptionEngine over the shared OpcConnectionPool. The OPC client certificate now always comes from MinIO + Mongo (CertProvider, Basic256Sha256) — no more certs/ shipped in the Gitea repository, so a new version branch no longer breaks the reads. - io/slots.py: parse_slot accepts slot v2 (names[]) and converts the legacy v1 (tag_name + topics; cert_path => use_security, path dropped), so gateway and Orchestrator can be deployed in any order. - managers/server_reader.py: new per-server reader with the OpcManager interface the IngestorManager already drives (connect, create_subscription, subscribe, unsubscribe, check_cycles, check_opc_listenning, shutdown). - IngestorManager owns the read path: lazy-MinIO CertProvider, the pool and a read DataSink, built without touching the network. Slots from Redis are normalized on lease and refresh; a malformed slot is notified and the last good config is kept. - GatewayRuntime borrows the pool (writes/gRPC) instead of building it, and no longer closes it on stop: a Postgres/NATS failure can't stop the reads. - pool: pin/unpin per active subscription so release_idle never evicts a subscribed connection; public evict() for lost servers / changed config. - SubscriptionEngine pins on subscribe, unpins on unsubscribe, and drops a subscription locally even when the server-side delete fails. - DataSink: model_id only written when the name has one, keeping v1 documents identical to the legacy ones. - config: CERT_BUCKET default is the legacy streamlit-connectors bucket (D7). - Removes OpcManager, its tests and scripts/generate-and-push-opc-cert.sh; README/CLAUDE.md document the MinIO certificate and the slot v2 format. Tests: 218 unit tests pass; ruff check/format clean. Local end-to-end against an asyncua server: v1 slot with two topics -> one doc per collection, legacy doc shape, idle sweep keeps the subscription, lost-server shutdown and re-initialization work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
76 lines
5.9 KiB
Markdown
76 lines
5.9 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project
|
|
|
|
OPC UA data ingestion service. Pulls tags from OPC UA servers, streams to Kafka (optional) and persists to MongoDB. Horizontally scaled via Redis-coordinated slot leasing — multiple pods share tag load without central orchestration.
|
|
|
|
Depends on internal shared lib `sientia-dataops-library` (imported as `sientia_do`, pinned in `requirements.txt` via git+ssh) for logging, notifications, metrics controller, and Mongo/Redis repositories. Base classes/utilities from `sientia_do` won't be visible in this repo — check that lib's source if behavior there is unclear.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
python3.11 -m venv venv && source ./venv/bin/activate
|
|
pip install -r requirements.txt -r requirements-dev.txt
|
|
|
|
./run_local.sh # loads .env, runs python -m ingestor.app
|
|
./run_coverage.sh # pytest --cov=ingestor --cov-report=html, opens report
|
|
|
|
pytest # all tests
|
|
pytest tests/unit # unit only
|
|
pytest tests/functional # functional only (spins up real-ish clients, see conftest.py)
|
|
pytest tests/unit/test_ingestor.py::TestClass::test_name # single test
|
|
pytest --cov=ingestor --cov-report=html # coverage report -> htmlcov/
|
|
|
|
ruff check . # lint
|
|
ruff format . # format
|
|
mypy ingestor # type check
|
|
bandit -r ingestor # security scan
|
|
```
|
|
|
|
External deps (Redis, MongoDB, Kafka) must be reachable — via `kubectl port-forward`, local docker-compose, or cloud services. Config lives in `.env` (copy from `.env.example`).
|
|
|
|
## Architecture
|
|
|
|
Manager-based pipeline, one manager per concern, composed by `IngestorManager`:
|
|
|
|
```
|
|
app.py (asyncio loop, signals, Prometheus server)
|
|
-> Ingestor (ingestor.py) — lifecycle: prepare_ingestor() -> loop() -> shutdown()
|
|
-> IngestorManager (managers/ingestor_manager.py) — central coordinator
|
|
-> ResourceManager (managers/resource_manager.py) — Redis slot leases, heartbeats, active-ingestor registry
|
|
-> ServerReader (managers/server_reader.py), one per OPC server — SubscriptionEngine over the shared OpcConnectionPool
|
|
-> OpcConnectionPool (connection/pool.py) + CertProvider (connection/certs.py) — pooled connections, cert from MinIO
|
|
-> DataManager (managers/data_manager.py) — MongoDB persistence + optional Kafka publish
|
|
```
|
|
|
|
**Slot model**: OPC tag configs live in Redis under keys like `slot:opc_tags:<n>`, each holding one or more OPC servers and their tags (see README "OPC Server Configuration" for the JSON shape). A "slot" is the unit of lease/assignment; each ingestor pod leases some number of slots via `ResourceManager` and only subscribes to the tags in slots it holds.
|
|
|
|
**Main loop** (`Ingestor.loop()` in `ingestor.py`) each cycle:
|
|
1. `declare_active()` + heartbeat (keeps this pod visible to peers via Redis TTL keys — `LEASE_TTL`/`HEARTBEAT_TTL`)
|
|
2. Reads current active-ingestor count, lease count, slot count
|
|
3. `manage_no_slots()` — grabs one slot if this pod is idle and slots exist
|
|
4. `manage_leases()` — rebalances: acquires slots if other pods are lacking, drops extras (keeps at most 1 slot per pod when supply is sufficient) — this is the load-balancing algorithm
|
|
5. `update_slot_config()` + `check_opc_servers_integrity()` — refresh OPC configs, detect stalled connections
|
|
6. `update_ingestor_manager()` — diffs current vs previous managed tags, subscribes new, resubscribes changed, unsubscribes removed
|
|
|
|
Any change to slot/lease counts should be traced through `manage_leases`/`manage_no_slots` — that's where the rebalancing math lives, not in `ResourceManager` itself (which just does Redis primitives).
|
|
|
|
**ServerReader** (SIENTIAPDE-1974, replaced the legacy `OpcManager`) reads one OPC server through a `SubscriptionEngine`: the connection comes from the shared `OpcConnectionPool` (also used by NATS writes and gRPC), which fetches the platform certificate from MinIO via `CertProvider` and uses `SecurityPolicyBasic256Sha256` — no certificate files in the repo/image. Slots are parsed by `io/slots.py` (`parse_slot`), which accepts slot v2 (`names[]`) and converts the legacy v1 (`topics`). `datachange_notification` fans each value out to every `names` entry via `DataSink`. Subscribed connections are pinned in the pool so the idle sweep never drops them. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles` / `check_opc_listenning`); a lost server is shut down (pool connection evicted) and re-initialized on the next loop.
|
|
|
|
**Metrics** (`ingestor/metrics.py`): all Prometheus metrics defined here, always labeled by `pod_id`. Emitted via `SientiaMonitoring.emit_metric(...)` (from `sientia_do`), not the prometheus client directly — new metrics should follow that pattern for consistency with logging/notifications.
|
|
|
|
**Error handling convention**: broad `except Exception` + `traceback.print_exc()` + notification-handler alert is intentional throughout (see `ruff.lint.ignore` for `BLE001` in `pyproject.toml`) — this is an always-on service where a single tag/server failure must not kill the pod; failures are surfaced via notifications/metrics instead of raised.
|
|
|
|
## Config
|
|
|
|
Env vars documented in README ("Configuration" section) and `.env.example`. Notable ones affecting behavior: `LEASE_TTL`/`HEARTBEAT_TTL` (failover speed vs stability), `POLL_INTERVAL` (loop cadence), `EXPORT_TO_KAFKA`.
|
|
|
|
## Testing notes
|
|
|
|
- `tests/unit/managers/` mirrors `ingestor/managers/` 1:1.
|
|
- `tests/functional/` exercises more end-to-end paths against `conftest.py` fixtures.
|
|
- Coverage config (`pyproject.toml` / `.coveragerc`) excludes `ingestor/app.py` (thin entrypoint, signal wiring) — don't chase coverage there.
|
|
- `ruff` complexity cap is `max-complexity = 15` (mccabe) — factor out branches in manager methods before that.
|