Commit Graph

138 Commits

Author SHA1 Message Date
PedroHMCosme
54408d6160 fix(gateway): suppress bandit B608 false positive in update_write_status
schema/table are allowlist-validated via _validate_identifier() before
interpolation (see docstring above), not a real SQL injection vector.
The existing '# noqa: S608' only silences ruff; bandit runs as its own
CI step and needs its own '# nosec B608' suppression syntax.
2026-09-22 15:59:36 -03:00
PedroHMCosme
c0fd5c1b50 Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 12:03:29 -03:00
PedroHMCosme
e76df05e7a fix(gateway): suppress SonarQube S7504 false positive in close_all
list() here isn't redundant: _evict() pops from self._connections, so
iterating the dict directly would raise 'changed size during iteration'.
NOSONAR + comment explaining why, per Sonar's own suppression convention
already used in ingestor/app.py.
2026-09-22 12:03:21 -03:00
PedroHMCosme
673b13cc6e fix(gateway): adapt DataValue construction to asyncua 2.x
main's requirements.txt unpins asyncua (PR #30), so CI installs 2.0.1,
where DataValue's field was renamed StatusCode_ -> StatusCode. The old
kwarg made every OPC write raise TypeError at runtime - masked locally
by a stale asyncua 1.0.6 in the dev conda env. Also uses ua.DateTime
(a datetime subclass) and ua.UInt32 to satisfy the new stricter types,
with no behavior change (verified via binary encode/decode round-trip).
2026-09-22 12:02:29 -03:00
PedroHMCosme
4f4037b4ae fix(gateway): satisfy mypy on nats/grpc adapters and generated proto code
- nats_consumer/grpc_server: annotate nc/sub/server attrs (were inferred
  as None-only from __init__, breaking on the real assignment later).
- active_read: narrow on 'if names:' instead of the derived 'persisted'
  bool so mypy sees names as non-Optional inside the publish_reading call.
- pyproject: exclude ingestor/proto/ (protoc/grpcio-tools output) from
  mypy's file walk and skip following it as an import - it's generated,
  not hand-maintained, and mypy can't see its runtime-built message classes.
2026-09-22 11:42:24 -03:00
PedroHMCosme
a215523fef Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 11:33:48 -03:00
PedroHMCosme
cfa8078334 fix(gateway): don't clobber asyncua's default application_uri with None
Only set application_uri when server_uri is configured; otherwise
asyncua's sane default ('urn:freeopcua:client') stood, but the code
explicitly overwrote it with None. Also fixes mypy strict-optional
error caught by the quality gate.
2026-09-22 11:33:24 -03:00
PedroHMCosme
6f4b04e8c0 style: ruff format (new quality gate check picked up from main) 2026-09-22 11:27:55 -03:00
PedroHMCosme
05732c66b8 Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 11:26:37 -03:00
PedroHMCosme
cdf7521530 style: ruff format (new quality gate check picked up from main) 2026-09-22 11:26:26 -03:00
PedroHMCosme
61d8f8da9c Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 10:48:44 -03:00
PedroHMCosme
3dcc59f65e Merge remote-tracking branch 'origin/main' into feature/SIENTIAPDE-1972-gateway-foundations
# Conflicts:
#	ingestor/ingestor.py
#	ingestor/managers/data_manager.py
#	requirements.txt
#	tests/unit/managers/test_data_manager.py
2026-09-22 10:47:13 -03:00
810bf54d56 fix(ingestor): Remove __del__ methods for async shutdown 2026-08-05 15:25:06 -03:00
a239a1a706 SAM0123-231: Enhance OPC UA client setup by asserting certificate and private key presence and assigning pod ID to client description. 2026-07-23 16:18:28 -03:00
b10c226e4e SAM0123-231: Remove Kafka integration from Ingestor service. 2026-07-23 12:25:29 -03:00
c157d1649a SAM0123-231: Broaden quality gate workflow to include release and feature branches; correct asynchronous call for OPC server updates in ingestor. 2026-07-23 12:04:13 -03:00
PedroHMCosme
52223e6c09 feat(gateway): gRPC server ReadTag (C4) 2026-07-15 08:39:54 -03:00
PedroHMCosme
1798b69c8b feat(gateway): gRPC proto + generated stubs (C4) 2026-07-15 08:36:08 -03:00
PedroHMCosme
99a715cc16 feat(gateway): ActiveReadEngine (scheduled + read_now) 2026-07-14 22:30:06 -03:00
PedroHMCosme
f7f356fbd5 refactor(gateway): ponytail F2 - drop dead metadata, unreachable variant fallback, redundant except member 2026-07-14 22:24:02 -03:00
PedroHMCosme
165ac79783 feat(gateway): NatsWriteConsumer (Core NATS, queue group) (C2) 2026-07-14 21:31:53 -03:00
PedroHMCosme
abff80e0d3 fix(gateway): tighten SQL identifier validation to fullmatch (reject trailing newline) 2026-07-14 19:31:11 -03:00
PedroHMCosme
905da5151b feat(gateway): DataSink.update_write_status Postgres by timestamp+model_id (C3); drop unused metadata param 2026-07-14 19:24:24 -03:00
PedroHMCosme
86d0f6cf23 fix(gateway): make WriteEngine + converter raise-safe against malformed write commands 2026-07-14 19:18:52 -03:00
PedroHMCosme
dcf8369638 feat(gateway): WriteEngine (status_code/timestamp/flags, safe converter, rate-limit) 2026-07-14 16:59:43 -03:00
PedroHMCosme
00c74e5606 feat(gateway): SubscriptionEngine multi-name fan-out (C1) 2026-07-14 16:17:13 -03:00
PedroHMCosme
323bf24a0f refactor(gateway): remove legacy Kafka egress (MongoDB-only) 2026-07-14 15:58:55 -03:00
PedroHMCosme
ae88ac662e feat(gateway): DataSink.publish_reading multi-name fan-out 2026-07-14 10:51:52 -03:00
PedroHMCosme
44df022e32 feat(gateway): slot v2 loader (contract C1) 2026-07-14 10:47:33 -03:00
PedroHMCosme
b57be9f5fe refactor(gateway): drop dead error_stack and unused metadata from OpcConnectionPool (ponytail) 2026-07-14 10:06:59 -03:00
PedroHMCosme
0b29857089 fix(gateway): per-server lock in OpcConnectionPool to avoid duplicate connects 2026-07-14 09:50:14 -03:00
PedroHMCosme
5d30a5c547 feat(gateway): OpcConnectionPool (per-server, reuse, idle-evict) 2026-07-14 09:35:53 -03:00
PedroHMCosme
ec2a324a84 feat(gateway): cert provider from MinIO+Mongo (sientia-dataops) 2026-07-13 19:20:41 -03:00
PedroHMCosme
b05c2fd76a feat(gateway): Basic256Sha256 + channel>session lifetime (1879) 2026-07-13 18:47:33 -03:00
PedroHMCosme
d6cb1e492d feat(gateway): monotonic clock helper (1879) 2026-07-13 18:39:23 -03:00
PedroHMCosme
55f67216ee feat(gateway): safe type converter (1879 BUG-1/BUG-2) 2026-07-13 18:31:07 -03:00
PedroHMCosme
3dd4f60b0e feat(gateway): add write/active-read/error metrics 2026-07-13 18:27:30 -03:00
89278827e7 SIENTIAPDE-1325
Enhance release workflow to trigger only on merged pull requests and streamline data insertion in DataManager

- Updated the release workflow to execute only when a pull request is merged.
- Simplified the document insertion logic in DataManager by consolidating the dictionary construction.
- Refactored OPC Manager tests to utilize asynchronous mocks for improved accuracy in testing.
2025-11-10 09:14:35 -03:00
373245d5a1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics immediately after connection attempt

- Moved the emission of OPC_CONNECTIONS_TOTAL metric to occur right after the connection attempt, ensuring accurate tracking of connection status.
- Removed redundant metric emission logic that was previously placed after a successful connection.
2025-11-07 09:56:48 -03:00
032be28eb1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics after successful connection establishment

- Moved the metric emission for OPC_CONNECTIONS_TOTAL to occur after a successful connection.
- Removed redundant metric increment logic to streamline the connection process.
2025-11-07 08:58:11 -03:00
1cb687aa37 SIENTIAPDE-1325
Update metrics.py to reorganize OPC Manager metrics section

- Added a comment header for OPC Manager metrics for better code organization.
- Removed redundant comment header to streamline the code structure.
2025-11-07 08:49:09 -03:00
c4f8fe95a8 SIENTIAPDE-1325
Update IngestorManager to use asynchronous methods for checking OPC server cycles and listening status
2025-11-06 16:40:45 -03:00
254be06004 SIENTIAPDE-1325
Update requirements.txt to use version 1.5.3 of sientia-dataops-library and enhance DataManager and ResourceManager with improved logging and timestamping for data insertion and retrieval.
2025-11-05 12:38:35 -03:00
d93c497914 SIENTIAPDE-1325
Enhance Ingestor Class with Asynchronous Metrics and Monitoring Integration

- Refactored Ingestor class to inherit from SientiaMonitoring for improved observability.
- Updated methods to utilize asynchronous operations for declaring active ingestors and managing slots.
- Integrated pod_id label into various metrics for better tracking.
- Adjusted unit tests to reflect changes in the Ingestor initialization and asynchronous behavior.
2025-11-03 10:33:05 -03:00
dfd08e883f SIENTIAPDE-1325
Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations

- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
2025-11-03 09:41:49 -03:00
85a371a38b SIENTIAPDE-1325
SIENTIAPDE-1325: Refactor Ingestor and Manager Classes for Enhanced Asynchronous Operations

- Introduced asynchronous methods across Ingestor, IngestorManager, DataManager, and OpcManager to improve performance and responsiveness.
- Integrated MetricsController into various classes for better observability and monitoring.
- Updated Redis and MongoDB interactions to support asynchronous operations, enhancing data handling efficiency.
- Removed deprecated Redis metrics and streamlined resource management logic.
- Adjusted unit tests to accommodate the new asynchronous behavior and ensure proper mocking of async methods.
2025-10-31 16:32:45 -03:00
8e3d62ff76 SIENTIAPDE-1314
Refactor IngestorManager to streamline server disconnection process

- Simplified the logic in the `update_opc_servers` method by directly appending servers to the disconnection list and utilizing the new `remove_server` method for handling server shutdowns.
- Removed redundant code related to managing the `managed_tags` during server disconnection, enhancing code clarity and maintainability.
2025-10-23 15:17:47 -03:00
56023dfdf4 SIENTIAPDE-1314
Refactor IngestorManager and OpcManager for improved server management

- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
2025-10-23 15:11:32 -03:00
83e43edf95 Fix formatting of NOSONAR comment in update_opc_servers method for consistency 2025-10-23 11:58:00 -03:00
f415207d5b Update OPC server update method to include NOSONAR comment for code quality tools 2025-10-23 11:55:55 -03:00