Swaps the legacy OpcManager read path for the gateway's SubscriptionEngine over
the shared OpcConnectionPool. The OPC client certificate now always comes from
MinIO + Mongo (CertProvider, Basic256Sha256) — no more certs/ shipped in the
Gitea repository, so a new version branch no longer breaks the reads.
- io/slots.py: parse_slot accepts slot v2 (names[]) and converts the legacy
v1 (tag_name + topics; cert_path => use_security, path dropped), so gateway
and Orchestrator can be deployed in any order.
- managers/server_reader.py: new per-server reader with the OpcManager
interface the IngestorManager already drives (connect, create_subscription,
subscribe, unsubscribe, check_cycles, check_opc_listenning, shutdown).
- IngestorManager owns the read path: lazy-MinIO CertProvider, the pool and a
read DataSink, built without touching the network. Slots from Redis are
normalized on lease and refresh; a malformed slot is notified and the last
good config is kept.
- GatewayRuntime borrows the pool (writes/gRPC) instead of building it, and no
longer closes it on stop: a Postgres/NATS failure can't stop the reads.
- pool: pin/unpin per active subscription so release_idle never evicts a
subscribed connection; public evict() for lost servers / changed config.
- SubscriptionEngine pins on subscribe, unpins on unsubscribe, and drops a
subscription locally even when the server-side delete fails.
- DataSink: model_id only written when the name has one, keeping v1 documents
identical to the legacy ones.
- config: CERT_BUCKET default is the legacy streamlit-connectors bucket (D7).
- Removes OpcManager, its tests and scripts/generate-and-push-opc-cert.sh;
README/CLAUDE.md document the MinIO certificate and the slot v2 format.
Tests: 218 unit tests pass; ruff check/format clean. Local end-to-end against
an asyncua server: v1 slot with two topics -> one doc per collection, legacy
doc shape, idle sweep keeps the subscription, lost-server shutdown and
re-initialization work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the dependency fix from #39 (grpcio-tools moved to
requirements-dev.txt, protobuf>=7.35 added) and the 2.0.0-rc2 version.
Conflict in requirements-dev.txt resolved keeping both sides:
python-dotenv (rc/ingestor2) and grpcio-tools (release). Also restores
the missing newline that had glued python-dotenv onto the ipdb comment
line, so pip was not installing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workers install only from the cluster package index (install-entrypoint.sh:
pip install --index-url $PYPI_SERVER). The 2.0.0-rc1 deploy never became Ready
because two new requirements are not mirrored there:
- grpcio-tools: only needed to regenerate ingestor/proto/*_pb2*.py, which are
committed. Moved to requirements-dev.txt (nothing imports grpc_tools).
- nats-py: needed at runtime; mirrored by library-distribution-controller#92.
Also declares protobuf>=7.35 explicitly: the committed stubs were generated with
protobuf 7.35.0 and validate the runtime version on import; until now it only
arrived transitively. The mirror carries 7.36.x.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- data_sink.py: _IDENTIFIER_RE uses \w+/re.ASCII instead of the
equivalent-but-verbose [A-Za-z0-9_]+ character class (verified same
match behavior). NOSONAR on update_write_status's 'async with no
await inside' - deliberate, per the ponytail comment already in the
method body (uniform async interface across DataSink, no async pg driver).
- test_grpc_server.py: move make_request() out of the pytest.raises
block so exactly one call can raise the expected exception.
- sonar-project.properties: exclude ingestor/proto/ (generated grpc
code) from analysis - its ReadTag/etc. names are dictated by
grpcio-tools, not a style choice, and renaming them breaks the wire.
'/tmp/opc-certs' is a spec'd, overridable-via-env default (already
noqa'd for ruff's S108); bandit needs its own '# nosec B108' since it
runs as a separate CI step and doesn't read ruff's noqa comments.
schema/table are allowlist-validated via _validate_identifier() before
interpolation (see docstring above), not a real SQL injection vector.
The existing '# noqa: S608' only silences ruff; bandit runs as its own
CI step and needs its own '# nosec B608' suppression syntax.
list() here isn't redundant: _evict() pops from self._connections, so
iterating the dict directly would raise 'changed size during iteration'.
NOSONAR + comment explaining why, per Sonar's own suppression convention
already used in ingestor/app.py.
main's requirements.txt unpins asyncua (PR #30), so CI installs 2.0.1,
where DataValue's field was renamed StatusCode_ -> StatusCode. The old
kwarg made every OPC write raise TypeError at runtime - masked locally
by a stale asyncua 1.0.6 in the dev conda env. Also uses ua.DateTime
(a datetime subclass) and ua.UInt32 to satisfy the new stricter types,
with no behavior change (verified via binary encode/decode round-trip).
- nats_consumer/grpc_server: annotate nc/sub/server attrs (were inferred
as None-only from __init__, breaking on the real assignment later).
- active_read: narrow on 'if names:' instead of the derived 'persisted'
bool so mypy sees names as non-Optional inside the publish_reading call.
- pyproject: exclude ingestor/proto/ (protoc/grpcio-tools output) from
mypy's file walk and skip following it as an import - it's generated,
not hand-maintained, and mypy can't see its runtime-built message classes.
Only set application_uri when server_uri is configured; otherwise
asyncua's sane default ('urn:freeopcua:client') stood, but the code
explicitly overwrote it with None. Also fixes mypy strict-optional
error caught by the quality gate.
- **Quality Gate (.github/workflows/quality-gate.yml):**
- Expanded push triggers to include main, release/**, and feature/** branches.
- Added concurrency control to cancel in-progress runs, preventing redundant checks.
- Refined workflow permissions to be more specific (contents: read, pull-requests: write, issues: write), enhancing security.
- **Local Validation (validate.sh):**
- Removed the validate.sh script, as its checks are now integrated into or superseded by the CI pipeline.
- **AI Guidance (CLAUDE.md):**
- Introduced CLAUDE.md to provide comprehensive guidance for Claude Code, detailing project architecture, commands, and conventions for AI assistants.
- **SonarQube Configuration (sonar-project.properties):**
- Removed hardcoded sonar.projectVersion.
