SIENTIAPDE-2202 Add gateway end-to-end test suite (e2e/)
End-to-end suite at the repo root, following the Laborious/Orchestrator
pattern: numbered scenarios per topic in e2e/scenarios.md (1:1 with
test_scenario_* functions), scenario inputs as JSON, Postgres DDL mirrored
from production, `integration` marker, excluded from the default pytest run.
Every dependency runs for real: MongoDB, Redis, PostgreSQL, MinIO and NATS
via testcontainers, OPC UA servers in process (asyncua). Pods are built from
the environment like ingestor/app.py and driven one loop iteration at a time.
69 scenarios across 7 topics: slot leasing and failover, subscription reads,
NATS writes and write_status (incl. the SIENTIAPDE-2202 timestamp/quality
matrix, checked on the server and on the wire), gRPC ReadTag, active reads,
client certificates/security, pod lifecycle.
Replaces tests/functional (legacy slot v1 tests, fully commented out) and
tests/functional/test_write_e2e.py (now scenarios 3.1.x).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-2202-escrita-opc-timestamp-qualidade: Enable granular control over OPC UA SourceTimestamp and StatusCode in write commands. This implements write_timestamp and write_quality flags (command > env > default true) for DataValue construction. Upgraded asyncua to 2.0+ to fix StatusCode field naming. Enhanced README.md with new gateway environment variables and detailed write command behavior. Added comprehensive unit and E2E tests for this feature, and removed obsolete planning documents.
SIENTIAPDE-1973: Implement per-connector OPC client certificates and remove legacy read engine.
This update overhauls certificate management, transitioning from a single platform certificate to individual client certificates defined in the slot configuration. The CertProvider now downloads certificates from MinIO based on certificate_id, applying dynamic security policies (mode, policy) from the slot. The legacy READ_ENGINE=legacy option and OpcManager have been entirely removed, standardizing on the ServerReader with strict slot v2 enforcement. Write command resolution (ServerConfigResolver) now consistently uses connectors joined with opc_certificates. The CERT_BUCKET environment variable is no longer used.
The Laborious no longer records OPC outcomes (it inserts the row as
'pending' and only publishes the commands), so the comments and docs that
asked to keep a Laborious copy of the UPDATE in sync are updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GW1: write commands (C2) carry only the server_id; ServerConfigResolver
resolves the connection (command override, live pooled connection,
connectors, opc-servers) with a 60 s cache and a 10 s negative cache.
- GW2: write_status keeps the worst outcome (ok < error < session_bad,
pending/NULL replaced), same rule as the Laborious, built with SQLAlchemy
Core (no SQL formatting).
- GW3: update_write_status never raises; outcome (updated, kept,
row_not_found, invalid, error) logged and counted
(opc_write_status_update_total); runs off the event loop.
- GW4: the reason of a failed write is logged; opc_write_status_by_model
shows the last outcome; a bad timestamp is a failed write, not an
exception.
- GW5: the NATS first connect gives up after 5 s (nats-py's defaults block
the main loop for about 4 minutes while NATS is down); reconnection after
it is unbounded; idle_sweep restarts a connection closed for good; stop()
is safe on a closed connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rc3 OpcManager is restored unchanged (with its tests and
scripts/generate-and-push-opc-cert.sh) for validation and as a fallback until
the end of SIENTIAPDE-1974; it is removed only once it is unused and the test
suite passes the same without it.
- READ_ENGINE=v2 (default): ServerReader/SubscriptionEngine over the shared
pool, certificate from MinIO. READ_ENGINE=legacy: the rc3 OpcManager with
its own connection, slot v1 only (a v2 slot is rejected and notified) and
the certificate files of the Gitea clone. An invalid value falls back to v2
with a warning. Writes/gRPC always use the pool.
- pool: OPC clients carry the pod id as application/session name and product
URI (as OpcManager did), so each pod's session is recognizable in the
server console; application_uri stays the server_uri (certificate SAN).
- README/CLAUDE.md document the plan B and how to remove it.
Tests: 275 unit tests pass (restored OpcManager tests unchanged); ruff clean.
Local e2e: both engines read the same slot v1 from an asyncua server and write
identical documents to the same collections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Swaps the legacy OpcManager read path for the gateway's SubscriptionEngine over
the shared OpcConnectionPool. The OPC client certificate now always comes from
MinIO + Mongo (CertProvider, Basic256Sha256) — no more certs/ shipped in the
Gitea repository, so a new version branch no longer breaks the reads.
- io/slots.py: parse_slot accepts slot v2 (names[]) and converts the legacy
v1 (tag_name + topics; cert_path => use_security, path dropped), so gateway
and Orchestrator can be deployed in any order.
- managers/server_reader.py: new per-server reader with the OpcManager
interface the IngestorManager already drives (connect, create_subscription,
subscribe, unsubscribe, check_cycles, check_opc_listenning, shutdown).
- IngestorManager owns the read path: lazy-MinIO CertProvider, the pool and a
read DataSink, built without touching the network. Slots from Redis are
normalized on lease and refresh; a malformed slot is notified and the last
good config is kept.
- GatewayRuntime borrows the pool (writes/gRPC) instead of building it, and no
longer closes it on stop: a Postgres/NATS failure can't stop the reads.
- pool: pin/unpin per active subscription so release_idle never evicts a
subscribed connection; public evict() for lost servers / changed config.
- SubscriptionEngine pins on subscribe, unpins on unsubscribe, and drops a
subscription locally even when the server-side delete fails.
- DataSink: model_id only written when the name has one, keeping v1 documents
identical to the legacy ones.
- config: CERT_BUCKET default is the legacy streamlit-connectors bucket (D7).
- Removes OpcManager, its tests and scripts/generate-and-push-opc-cert.sh;
README/CLAUDE.md document the MinIO certificate and the slot v2 format.
Tests: 218 unit tests pass; ruff check/format clean. Local end-to-end against
an asyncua server: v1 slot with two topics -> one doc per collection, legacy
doc shape, idle sweep keeps the subscription, lost-server shutdown and
re-initialization work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enhance documentation and improve shutdown methods for DataManager and IngestorManager
- Updated README.md to clarify persistent storage capabilities.
- Enhanced docstrings in DataManager and IngestorManager for shutdown processes, detailing resource cleanup and error handling.
- Improved comments in various methods to provide clearer explanations of functionality and side effects.
Update README.md to clarify data streaming and storage features
- Revised descriptions for Real-time Data Streaming and Persistent Storage to emphasize historical data streaming capabilities.
- Updated Data Manager section to reflect the integration of both MongoDB and Kafka for data management.
- Enhanced configuration example in the prerequisites section with additional fields for server configuration.
Remove deprecated files and enhance documentation
- Deleted `coverage.sh`, `docker-compose.yaml`, `Dockerfile`, and simulator-related files to streamline the project structure.
- Updated `README.md` to provide a comprehensive overview of the OPC Ingestor, including features, architecture, installation, usage, and troubleshooting.
- Enhanced docstrings across various classes and methods in the `ingestor` module for better clarity and maintainability.
- Improved Prometheus metrics documentation in `metrics.py` to ensure proper monitoring and observability of the system.
Refactor ingestor lease management and update Docker configurations
- Updated CMD in Dockerfile to use the correct application entry point.
- Added docker compose command to remove volumes in README.
- Removed unnecessary restart policy from ingestor service in docker-compose.yaml.
- Refactored manage_leases method in Ingestor class to improve clarity and parameter naming.
- Added get_number_of_leases method in IngestorManager to retrieve active leases.
- Implemented get_all_leases method in ResourceManager to fetch active leases from Redis.
- Enhanced unit tests for lease management in test_ingestor_manager and test_resource_manager.
Refactor OpcManager initialization and unsubscribe method; enhance logging
- Removed the `initialize_from_config` method from `OpcManager` class and adjusted the constructor to handle configuration directly.
- Improved the `unsubscribe` method to include detailed logging for non-existent subscriptions.
- Updated tests in `test_opc_manager.py` to reflect changes in the `OpcManager` class.
- Commented out Docker-related fixtures in `conftest.py` for potential future use.
- Enhanced test coverage in `test_single_node.py` and `test_data_manager.py` with additional scenarios and assertions.
- Introduced new methods in `Ingestor` and `IngestorManager` classes to manage server subscriptions and leases more effectively.
- Added new tests for `Ingestor` class to validate initialization and slot management logic.
- Implemented logging improvements across various classes to ensure better traceability of actions and errors.
Enhance README and Docker setup; improve IngestorManager and DataManager functionality, add error handling, and implement unit tests for OPC initialization and subscription management.
