Commit Graph

137 Commits

Author SHA1 Message Date
PedroHMCosme
c0fd5c1b50 Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 12:03:29 -03:00
PedroHMCosme
e76df05e7a fix(gateway): suppress SonarQube S7504 false positive in close_all
list() here isn't redundant: _evict() pops from self._connections, so
iterating the dict directly would raise 'changed size during iteration'.
NOSONAR + comment explaining why, per Sonar's own suppression convention
already used in ingestor/app.py.
2026-09-22 12:03:21 -03:00
PedroHMCosme
673b13cc6e fix(gateway): adapt DataValue construction to asyncua 2.x
main's requirements.txt unpins asyncua (PR #30), so CI installs 2.0.1,
where DataValue's field was renamed StatusCode_ -> StatusCode. The old
kwarg made every OPC write raise TypeError at runtime - masked locally
by a stale asyncua 1.0.6 in the dev conda env. Also uses ua.DateTime
(a datetime subclass) and ua.UInt32 to satisfy the new stricter types,
with no behavior change (verified via binary encode/decode round-trip).
2026-09-22 12:02:29 -03:00
PedroHMCosme
4f4037b4ae fix(gateway): satisfy mypy on nats/grpc adapters and generated proto code
- nats_consumer/grpc_server: annotate nc/sub/server attrs (were inferred
  as None-only from __init__, breaking on the real assignment later).
- active_read: narrow on 'if names:' instead of the derived 'persisted'
  bool so mypy sees names as non-Optional inside the publish_reading call.
- pyproject: exclude ingestor/proto/ (protoc/grpcio-tools output) from
  mypy's file walk and skip following it as an import - it's generated,
  not hand-maintained, and mypy can't see its runtime-built message classes.
2026-09-22 11:42:24 -03:00
PedroHMCosme
a215523fef Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 11:33:48 -03:00
PedroHMCosme
cfa8078334 fix(gateway): don't clobber asyncua's default application_uri with None
Only set application_uri when server_uri is configured; otherwise
asyncua's sane default ('urn:freeopcua:client') stood, but the code
explicitly overwrote it with None. Also fixes mypy strict-optional
error caught by the quality gate.
2026-09-22 11:33:24 -03:00
PedroHMCosme
6f4b04e8c0 style: ruff format (new quality gate check picked up from main) 2026-09-22 11:27:55 -03:00
PedroHMCosme
05732c66b8 Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 11:26:37 -03:00
PedroHMCosme
cdf7521530 style: ruff format (new quality gate check picked up from main) 2026-09-22 11:26:26 -03:00
PedroHMCosme
61d8f8da9c Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 10:48:44 -03:00
PedroHMCosme
3dcc59f65e Merge remote-tracking branch 'origin/main' into feature/SIENTIAPDE-1972-gateway-foundations
# Conflicts:
#	ingestor/ingestor.py
#	ingestor/managers/data_manager.py
#	requirements.txt
#	tests/unit/managers/test_data_manager.py
2026-09-22 10:47:13 -03:00
810bf54d56 fix(ingestor): Remove __del__ methods for async shutdown 2026-08-05 15:25:06 -03:00
a239a1a706 SAM0123-231: Enhance OPC UA client setup by asserting certificate and private key presence and assigning pod ID to client description. 2026-07-23 16:18:28 -03:00
b10c226e4e SAM0123-231: Remove Kafka integration from Ingestor service. 2026-07-23 12:25:29 -03:00
c157d1649a SAM0123-231: Broaden quality gate workflow to include release and feature branches; correct asynchronous call for OPC server updates in ingestor. 2026-07-23 12:04:13 -03:00
PedroHMCosme
52223e6c09 feat(gateway): gRPC server ReadTag (C4) 2026-07-15 08:39:54 -03:00
PedroHMCosme
1798b69c8b feat(gateway): gRPC proto + generated stubs (C4) 2026-07-15 08:36:08 -03:00
PedroHMCosme
99a715cc16 feat(gateway): ActiveReadEngine (scheduled + read_now) 2026-07-14 22:30:06 -03:00
PedroHMCosme
f7f356fbd5 refactor(gateway): ponytail F2 - drop dead metadata, unreachable variant fallback, redundant except member 2026-07-14 22:24:02 -03:00
PedroHMCosme
165ac79783 feat(gateway): NatsWriteConsumer (Core NATS, queue group) (C2) 2026-07-14 21:31:53 -03:00
PedroHMCosme
abff80e0d3 fix(gateway): tighten SQL identifier validation to fullmatch (reject trailing newline) 2026-07-14 19:31:11 -03:00
PedroHMCosme
905da5151b feat(gateway): DataSink.update_write_status Postgres by timestamp+model_id (C3); drop unused metadata param 2026-07-14 19:24:24 -03:00
PedroHMCosme
86d0f6cf23 fix(gateway): make WriteEngine + converter raise-safe against malformed write commands 2026-07-14 19:18:52 -03:00
PedroHMCosme
dcf8369638 feat(gateway): WriteEngine (status_code/timestamp/flags, safe converter, rate-limit) 2026-07-14 16:59:43 -03:00
PedroHMCosme
00c74e5606 feat(gateway): SubscriptionEngine multi-name fan-out (C1) 2026-07-14 16:17:13 -03:00
PedroHMCosme
323bf24a0f refactor(gateway): remove legacy Kafka egress (MongoDB-only) 2026-07-14 15:58:55 -03:00
PedroHMCosme
ae88ac662e feat(gateway): DataSink.publish_reading multi-name fan-out 2026-07-14 10:51:52 -03:00
PedroHMCosme
44df022e32 feat(gateway): slot v2 loader (contract C1) 2026-07-14 10:47:33 -03:00
PedroHMCosme
b57be9f5fe refactor(gateway): drop dead error_stack and unused metadata from OpcConnectionPool (ponytail) 2026-07-14 10:06:59 -03:00
PedroHMCosme
0b29857089 fix(gateway): per-server lock in OpcConnectionPool to avoid duplicate connects 2026-07-14 09:50:14 -03:00
PedroHMCosme
5d30a5c547 feat(gateway): OpcConnectionPool (per-server, reuse, idle-evict) 2026-07-14 09:35:53 -03:00
PedroHMCosme
ec2a324a84 feat(gateway): cert provider from MinIO+Mongo (sientia-dataops) 2026-07-13 19:20:41 -03:00
PedroHMCosme
b05c2fd76a feat(gateway): Basic256Sha256 + channel>session lifetime (1879) 2026-07-13 18:47:33 -03:00
PedroHMCosme
d6cb1e492d feat(gateway): monotonic clock helper (1879) 2026-07-13 18:39:23 -03:00
PedroHMCosme
55f67216ee feat(gateway): safe type converter (1879 BUG-1/BUG-2) 2026-07-13 18:31:07 -03:00
PedroHMCosme
3dd4f60b0e feat(gateway): add write/active-read/error metrics 2026-07-13 18:27:30 -03:00
89278827e7 SIENTIAPDE-1325
Enhance release workflow to trigger only on merged pull requests and streamline data insertion in DataManager

- Updated the release workflow to execute only when a pull request is merged.
- Simplified the document insertion logic in DataManager by consolidating the dictionary construction.
- Refactored OPC Manager tests to utilize asynchronous mocks for improved accuracy in testing.
2025-11-10 09:14:35 -03:00
373245d5a1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics immediately after connection attempt

- Moved the emission of OPC_CONNECTIONS_TOTAL metric to occur right after the connection attempt, ensuring accurate tracking of connection status.
- Removed redundant metric emission logic that was previously placed after a successful connection.
2025-11-07 09:56:48 -03:00
032be28eb1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics after successful connection establishment

- Moved the metric emission for OPC_CONNECTIONS_TOTAL to occur after a successful connection.
- Removed redundant metric increment logic to streamline the connection process.
2025-11-07 08:58:11 -03:00
1cb687aa37 SIENTIAPDE-1325
Update metrics.py to reorganize OPC Manager metrics section

- Added a comment header for OPC Manager metrics for better code organization.
- Removed redundant comment header to streamline the code structure.
2025-11-07 08:49:09 -03:00
c4f8fe95a8 SIENTIAPDE-1325
Update IngestorManager to use asynchronous methods for checking OPC server cycles and listening status
2025-11-06 16:40:45 -03:00
254be06004 SIENTIAPDE-1325
Update requirements.txt to use version 1.5.3 of sientia-dataops-library and enhance DataManager and ResourceManager with improved logging and timestamping for data insertion and retrieval.
2025-11-05 12:38:35 -03:00
d93c497914 SIENTIAPDE-1325
Enhance Ingestor Class with Asynchronous Metrics and Monitoring Integration

- Refactored Ingestor class to inherit from SientiaMonitoring for improved observability.
- Updated methods to utilize asynchronous operations for declaring active ingestors and managing slots.
- Integrated pod_id label into various metrics for better tracking.
- Adjusted unit tests to reflect changes in the Ingestor initialization and asynchronous behavior.
2025-11-03 10:33:05 -03:00
dfd08e883f SIENTIAPDE-1325
Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations

- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
2025-11-03 09:41:49 -03:00
85a371a38b SIENTIAPDE-1325
SIENTIAPDE-1325: Refactor Ingestor and Manager Classes for Enhanced Asynchronous Operations

- Introduced asynchronous methods across Ingestor, IngestorManager, DataManager, and OpcManager to improve performance and responsiveness.
- Integrated MetricsController into various classes for better observability and monitoring.
- Updated Redis and MongoDB interactions to support asynchronous operations, enhancing data handling efficiency.
- Removed deprecated Redis metrics and streamlined resource management logic.
- Adjusted unit tests to accommodate the new asynchronous behavior and ensure proper mocking of async methods.
2025-10-31 16:32:45 -03:00
8e3d62ff76 SIENTIAPDE-1314
Refactor IngestorManager to streamline server disconnection process

- Simplified the logic in the `update_opc_servers` method by directly appending servers to the disconnection list and utilizing the new `remove_server` method for handling server shutdowns.
- Removed redundant code related to managing the `managed_tags` during server disconnection, enhancing code clarity and maintainability.
2025-10-23 15:17:47 -03:00
56023dfdf4 SIENTIAPDE-1314
Refactor IngestorManager and OpcManager for improved server management

- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
2025-10-23 15:11:32 -03:00
83e43edf95 Fix formatting of NOSONAR comment in update_opc_servers method for consistency 2025-10-23 11:58:00 -03:00
f415207d5b Update OPC server update method to include NOSONAR comment for code quality tools 2025-10-23 11:55:55 -03:00
eea6ccb48f Refactor OPC manager shutdown logic for improved reliability
Updated the IngestorManager class to ensure that OPC managers are properly shut down and removed from the management list when servers are lost or removed. This change enhances the integrity checks for OPC server management by ensuring asynchronous handling of shutdown operations.
2025-10-23 11:52:00 -03:00