0ac817a479
The rc3 OpcManager is restored unchanged (with its tests and scripts/generate-and-push-opc-cert.sh) for validation and as a fallback until the end of SIENTIAPDE-1974; it is removed only once it is unused and the test suite passes the same without it. - READ_ENGINE=v2 (default): ServerReader/SubscriptionEngine over the shared pool, certificate from MinIO. READ_ENGINE=legacy: the rc3 OpcManager with its own connection, slot v1 only (a v2 slot is rejected and notified) and the certificate files of the Gitea clone. An invalid value falls back to v2 with a warning. Writes/gRPC always use the pool. - pool: OPC clients carry the pod id as application/session name and product URI (as OpcManager did), so each pod's session is recognizable in the server console; application_uri stays the server_uri (certificate SAN). - README/CLAUDE.md document the plan B and how to remove it. Tests: 275 unit tests pass (restored OpcManager tests unchanged); ruff clean. Local e2e: both engines read the same slot v1 from an asyncua server and write identical documents to the same collections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
77 lines
6.2 KiB
Markdown
77 lines
6.2 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project
|
|
|
|
OPC UA data ingestion service. Pulls tags from OPC UA servers, streams to Kafka (optional) and persists to MongoDB. Horizontally scaled via Redis-coordinated slot leasing — multiple pods share tag load without central orchestration.
|
|
|
|
Depends on internal shared lib `sientia-dataops-library` (imported as `sientia_do`, pinned in `requirements.txt` via git+ssh) for logging, notifications, metrics controller, and Mongo/Redis repositories. Base classes/utilities from `sientia_do` won't be visible in this repo — check that lib's source if behavior there is unclear.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
python3.11 -m venv venv && source ./venv/bin/activate
|
|
pip install -r requirements.txt -r requirements-dev.txt
|
|
|
|
./run_local.sh # loads .env, runs python -m ingestor.app
|
|
./run_coverage.sh # pytest --cov=ingestor --cov-report=html, opens report
|
|
|
|
pytest # all tests
|
|
pytest tests/unit # unit only
|
|
pytest tests/functional # functional only (spins up real-ish clients, see conftest.py)
|
|
pytest tests/unit/test_ingestor.py::TestClass::test_name # single test
|
|
pytest --cov=ingestor --cov-report=html # coverage report -> htmlcov/
|
|
|
|
ruff check . # lint
|
|
ruff format . # format
|
|
mypy ingestor # type check
|
|
bandit -r ingestor # security scan
|
|
```
|
|
|
|
External deps (Redis, MongoDB, Kafka) must be reachable — via `kubectl port-forward`, local docker-compose, or cloud services. Config lives in `.env` (copy from `.env.example`).
|
|
|
|
## Architecture
|
|
|
|
Manager-based pipeline, one manager per concern, composed by `IngestorManager`:
|
|
|
|
```
|
|
app.py (asyncio loop, signals, Prometheus server)
|
|
-> Ingestor (ingestor.py) — lifecycle: prepare_ingestor() -> loop() -> shutdown()
|
|
-> IngestorManager (managers/ingestor_manager.py) — central coordinator
|
|
-> ResourceManager (managers/resource_manager.py) — Redis slot leases, heartbeats, active-ingestor registry
|
|
-> ServerReader (managers/server_reader.py), one per OPC server — SubscriptionEngine over the shared OpcConnectionPool
|
|
(READ_ENGINE=legacy: OpcManager (managers/opc_manager.py), the rc3 reader kept as plan B until the end of SIENTIAPDE-1974)
|
|
-> OpcConnectionPool (connection/pool.py) + CertProvider (connection/certs.py) — pooled connections, cert from MinIO
|
|
-> DataManager (managers/data_manager.py) — MongoDB persistence + optional Kafka publish
|
|
```
|
|
|
|
**Slot model**: OPC tag configs live in Redis under keys like `slot:opc_tags:<n>`, each holding one or more OPC servers and their tags (see README "OPC Server Configuration" for the JSON shape). A "slot" is the unit of lease/assignment; each ingestor pod leases some number of slots via `ResourceManager` and only subscribes to the tags in slots it holds.
|
|
|
|
**Main loop** (`Ingestor.loop()` in `ingestor.py`) each cycle:
|
|
1. `declare_active()` + heartbeat (keeps this pod visible to peers via Redis TTL keys — `LEASE_TTL`/`HEARTBEAT_TTL`)
|
|
2. Reads current active-ingestor count, lease count, slot count
|
|
3. `manage_no_slots()` — grabs one slot if this pod is idle and slots exist
|
|
4. `manage_leases()` — rebalances: acquires slots if other pods are lacking, drops extras (keeps at most 1 slot per pod when supply is sufficient) — this is the load-balancing algorithm
|
|
5. `update_slot_config()` + `check_opc_servers_integrity()` — refresh OPC configs, detect stalled connections
|
|
6. `update_ingestor_manager()` — diffs current vs previous managed tags, subscribes new, resubscribes changed, unsubscribes removed
|
|
|
|
Any change to slot/lease counts should be traced through `manage_leases`/`manage_no_slots` — that's where the rebalancing math lives, not in `ResourceManager` itself (which just does Redis primitives).
|
|
|
|
**ServerReader** (SIENTIAPDE-1974, default `READ_ENGINE=v2`; the legacy `OpcManager` stays as plan B with `READ_ENGINE=legacy` — slot v1 only, certificate files from the Gitea clone — until it is unused and the tests pass the same without it) reads one OPC server through a `SubscriptionEngine`: the connection comes from the shared `OpcConnectionPool` (also used by NATS writes and gRPC), which fetches the platform certificate from MinIO via `CertProvider` and uses `SecurityPolicyBasic256Sha256` — no certificate files in the repo/image. Slots are parsed by `io/slots.py` (`parse_slot`), which accepts slot v2 (`names[]`) and converts the legacy v1 (`topics`). `datachange_notification` fans each value out to every `names` entry via `DataSink`. Subscribed connections are pinned in the pool so the idle sweep never drops them. Tracks `non_receive_count` to detect silently-dead subscriptions (`check_cycles` / `check_opc_listenning`); a lost server is shut down (pool connection evicted) and re-initialized on the next loop.
|
|
|
|
**Metrics** (`ingestor/metrics.py`): all Prometheus metrics defined here, always labeled by `pod_id`. Emitted via `SientiaMonitoring.emit_metric(...)` (from `sientia_do`), not the prometheus client directly — new metrics should follow that pattern for consistency with logging/notifications.
|
|
|
|
**Error handling convention**: broad `except Exception` + `traceback.print_exc()` + notification-handler alert is intentional throughout (see `ruff.lint.ignore` for `BLE001` in `pyproject.toml`) — this is an always-on service where a single tag/server failure must not kill the pod; failures are surfaced via notifications/metrics instead of raised.
|
|
|
|
## Config
|
|
|
|
Env vars documented in README ("Configuration" section) and `.env.example`. Notable ones affecting behavior: `LEASE_TTL`/`HEARTBEAT_TTL` (failover speed vs stability), `POLL_INTERVAL` (loop cadence), `EXPORT_TO_KAFKA`.
|
|
|
|
## Testing notes
|
|
|
|
- `tests/unit/managers/` mirrors `ingestor/managers/` 1:1.
|
|
- `tests/functional/` exercises more end-to-end paths against `conftest.py` fixtures.
|
|
- Coverage config (`pyproject.toml` / `.coveragerc`) excludes `ingestor/app.py` (thin entrypoint, signal wiring) — don't chase coverage there.
|
|
- `ruff` complexity cap is `max-complexity = 15` (mccabe) — factor out branches in manager methods before that.
|