SIENTIAPDE-2202 Add gateway end-to-end test suite (e2e/)
End-to-end suite at the repo root, following the Laborious/Orchestrator
pattern: numbered scenarios per topic in e2e/scenarios.md (1:1 with
test_scenario_* functions), scenario inputs as JSON, Postgres DDL mirrored
from production, `integration` marker, excluded from the default pytest run.
Every dependency runs for real: MongoDB, Redis, PostgreSQL, MinIO and NATS
via testcontainers, OPC UA servers in process (asyncua). Pods are built from
the environment like ingestor/app.py and driven one loop iteration at a time.
69 scenarios across 7 topics: slot leasing and failover, subscription reads,
NATS writes and write_status (incl. the SIENTIAPDE-2202 timestamp/quality
matrix, checked on the server and on the wire), gRPC ReadTag, active reads,
client certificates/security, pod lifecycle.
Replaces tests/functional (legacy slot v1 tests, fully commented out) and
tests/functional/test_write_e2e.py (now scenarios 3.1.x).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-1973: Implement per-connector OPC client certificates and remove legacy read engine.
This update overhauls certificate management, transitioning from a single platform certificate to individual client certificates defined in the slot configuration. The CertProvider now downloads certificates from MinIO based on certificate_id, applying dynamic security policies (mode, policy) from the slot. The legacy READ_ENGINE=legacy option and OpcManager have been entirely removed, standardizing on the ServerReader with strict slot v2 enforcement. Write command resolution (ServerConfigResolver) now consistently uses connectors joined with opc_certificates. The CERT_BUCKET environment variable is no longer used.
The Laborious no longer records OPC outcomes (it inserts the row as
'pending' and only publishes the commands), so the comments and docs that
asked to keep a Laborious copy of the UPDATE in sync are updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GW1: write commands (C2) carry only the server_id; ServerConfigResolver
resolves the connection (command override, live pooled connection,
connectors, opc-servers) with a 60 s cache and a 10 s negative cache.
- GW2: write_status keeps the worst outcome (ok < error < session_bad,
pending/NULL replaced), same rule as the Laborious, built with SQLAlchemy
Core (no SQL formatting).
- GW3: update_write_status never raises; outcome (updated, kept,
row_not_found, invalid, error) logged and counted
(opc_write_status_update_total); runs off the event loop.
- GW4: the reason of a failed write is logged; opc_write_status_by_model
shows the last outcome; a bad timestamp is a failed write, not an
exception.
- GW5: the NATS first connect gives up after 5 s (nats-py's defaults block
the main loop for about 4 minutes while NATS is down); reconnection after
it is unbounded; idle_sweep restarts a connection closed for good; stop()
is safe on a closed connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rc3 OpcManager is restored unchanged (with its tests and
scripts/generate-and-push-opc-cert.sh) for validation and as a fallback until
the end of SIENTIAPDE-1974; it is removed only once it is unused and the test
suite passes the same without it.
- READ_ENGINE=v2 (default): ServerReader/SubscriptionEngine over the shared
pool, certificate from MinIO. READ_ENGINE=legacy: the rc3 OpcManager with
its own connection, slot v1 only (a v2 slot is rejected and notified) and
the certificate files of the Gitea clone. An invalid value falls back to v2
with a warning. Writes/gRPC always use the pool.
- pool: OPC clients carry the pod id as application/session name and product
URI (as OpcManager did), so each pod's session is recognizable in the
server console; application_uri stays the server_uri (certificate SAN).
- README/CLAUDE.md document the plan B and how to remove it.
Tests: 275 unit tests pass (restored OpcManager tests unchanged); ruff clean.
Local e2e: both engines read the same slot v1 from an asyncua server and write
identical documents to the same collections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Swaps the legacy OpcManager read path for the gateway's SubscriptionEngine over
the shared OpcConnectionPool. The OPC client certificate now always comes from
MinIO + Mongo (CertProvider, Basic256Sha256) — no more certs/ shipped in the
Gitea repository, so a new version branch no longer breaks the reads.
- io/slots.py: parse_slot accepts slot v2 (names[]) and converts the legacy
v1 (tag_name + topics; cert_path => use_security, path dropped), so gateway
and Orchestrator can be deployed in any order.
- managers/server_reader.py: new per-server reader with the OpcManager
interface the IngestorManager already drives (connect, create_subscription,
subscribe, unsubscribe, check_cycles, check_opc_listenning, shutdown).
- IngestorManager owns the read path: lazy-MinIO CertProvider, the pool and a
read DataSink, built without touching the network. Slots from Redis are
normalized on lease and refresh; a malformed slot is notified and the last
good config is kept.
- GatewayRuntime borrows the pool (writes/gRPC) instead of building it, and no
longer closes it on stop: a Postgres/NATS failure can't stop the reads.
- pool: pin/unpin per active subscription so release_idle never evicts a
subscribed connection; public evict() for lost servers / changed config.
- SubscriptionEngine pins on subscribe, unpins on unsubscribe, and drops a
subscription locally even when the server-side delete fails.
- DataSink: model_id only written when the name has one, keeping v1 documents
identical to the legacy ones.
- config: CERT_BUCKET default is the legacy streamlit-connectors bucket (D7).
- Removes OpcManager, its tests and scripts/generate-and-push-opc-cert.sh;
README/CLAUDE.md document the MinIO certificate and the slot v2 format.
Tests: 218 unit tests pass; ruff check/format clean. Local end-to-end against
an asyncua server: v1 slot with two topics -> one doc per collection, legacy
doc shape, idle sweep keeps the subscription, lost-server shutdown and
re-initialization work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- **Quality Gate (.github/workflows/quality-gate.yml):**
- Expanded push triggers to include main, release/**, and feature/** branches.
- Added concurrency control to cancel in-progress runs, preventing redundant checks.
- Refined workflow permissions to be more specific (contents: read, pull-requests: write, issues: write), enhancing security.
- **Local Validation (validate.sh):**
- Removed the validate.sh script, as its checks are now integrated into or superseded by the CI pipeline.
- **AI Guidance (CLAUDE.md):**
- Introduced CLAUDE.md to provide comprehensive guidance for Claude Code, detailing project architecture, commands, and conventions for AI assistants.
- **SonarQube Configuration (sonar-project.properties):**
- Removed hardcoded sonar.projectVersion.
