Commit Graph

131 Commits

Author SHA1 Message Date
PedroHMCosme
6f4b04e8c0 style: ruff format (new quality gate check picked up from main) 2026-09-22 11:27:55 -03:00
PedroHMCosme
05732c66b8 Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 11:26:37 -03:00
PedroHMCosme
cdf7521530 style: ruff format (new quality gate check picked up from main) 2026-09-22 11:26:26 -03:00
PedroHMCosme
61d8f8da9c Merge branch 'feature/SIENTIAPDE-1972-gateway-foundations' into feature/SIENTIAPDE-1972-gateway-write-read 2026-09-22 10:48:44 -03:00
PedroHMCosme
3dcc59f65e Merge remote-tracking branch 'origin/main' into feature/SIENTIAPDE-1972-gateway-foundations
# Conflicts:
#	ingestor/ingestor.py
#	ingestor/managers/data_manager.py
#	requirements.txt
#	tests/unit/managers/test_data_manager.py
2026-09-22 10:47:13 -03:00
810bf54d56 fix(ingestor): Remove __del__ methods for async shutdown 2026-08-05 15:25:06 -03:00
a239a1a706 SAM0123-231: Enhance OPC UA client setup by asserting certificate and private key presence and assigning pod ID to client description. 2026-07-23 16:18:28 -03:00
b10c226e4e SAM0123-231: Remove Kafka integration from Ingestor service. 2026-07-23 12:25:29 -03:00
c157d1649a SAM0123-231: Broaden quality gate workflow to include release and feature branches; correct asynchronous call for OPC server updates in ingestor. 2026-07-23 12:04:13 -03:00
PedroHMCosme
52223e6c09 feat(gateway): gRPC server ReadTag (C4) 2026-07-15 08:39:54 -03:00
PedroHMCosme
1798b69c8b feat(gateway): gRPC proto + generated stubs (C4) 2026-07-15 08:36:08 -03:00
PedroHMCosme
99a715cc16 feat(gateway): ActiveReadEngine (scheduled + read_now) 2026-07-14 22:30:06 -03:00
PedroHMCosme
f7f356fbd5 refactor(gateway): ponytail F2 - drop dead metadata, unreachable variant fallback, redundant except member 2026-07-14 22:24:02 -03:00
PedroHMCosme
165ac79783 feat(gateway): NatsWriteConsumer (Core NATS, queue group) (C2) 2026-07-14 21:31:53 -03:00
PedroHMCosme
abff80e0d3 fix(gateway): tighten SQL identifier validation to fullmatch (reject trailing newline) 2026-07-14 19:31:11 -03:00
PedroHMCosme
905da5151b feat(gateway): DataSink.update_write_status Postgres by timestamp+model_id (C3); drop unused metadata param 2026-07-14 19:24:24 -03:00
PedroHMCosme
86d0f6cf23 fix(gateway): make WriteEngine + converter raise-safe against malformed write commands 2026-07-14 19:18:52 -03:00
PedroHMCosme
dcf8369638 feat(gateway): WriteEngine (status_code/timestamp/flags, safe converter, rate-limit) 2026-07-14 16:59:43 -03:00
PedroHMCosme
00c74e5606 feat(gateway): SubscriptionEngine multi-name fan-out (C1) 2026-07-14 16:17:13 -03:00
PedroHMCosme
323bf24a0f refactor(gateway): remove legacy Kafka egress (MongoDB-only) 2026-07-14 15:58:55 -03:00
PedroHMCosme
ae88ac662e feat(gateway): DataSink.publish_reading multi-name fan-out 2026-07-14 10:51:52 -03:00
PedroHMCosme
44df022e32 feat(gateway): slot v2 loader (contract C1) 2026-07-14 10:47:33 -03:00
PedroHMCosme
b57be9f5fe refactor(gateway): drop dead error_stack and unused metadata from OpcConnectionPool (ponytail) 2026-07-14 10:06:59 -03:00
PedroHMCosme
0b29857089 fix(gateway): per-server lock in OpcConnectionPool to avoid duplicate connects 2026-07-14 09:50:14 -03:00
PedroHMCosme
5d30a5c547 feat(gateway): OpcConnectionPool (per-server, reuse, idle-evict) 2026-07-14 09:35:53 -03:00
PedroHMCosme
ec2a324a84 feat(gateway): cert provider from MinIO+Mongo (sientia-dataops) 2026-07-13 19:20:41 -03:00
PedroHMCosme
b05c2fd76a feat(gateway): Basic256Sha256 + channel>session lifetime (1879) 2026-07-13 18:47:33 -03:00
PedroHMCosme
d6cb1e492d feat(gateway): monotonic clock helper (1879) 2026-07-13 18:39:23 -03:00
PedroHMCosme
55f67216ee feat(gateway): safe type converter (1879 BUG-1/BUG-2) 2026-07-13 18:31:07 -03:00
PedroHMCosme
3dd4f60b0e feat(gateway): add write/active-read/error metrics 2026-07-13 18:27:30 -03:00
89278827e7 SIENTIAPDE-1325
Enhance release workflow to trigger only on merged pull requests and streamline data insertion in DataManager

- Updated the release workflow to execute only when a pull request is merged.
- Simplified the document insertion logic in DataManager by consolidating the dictionary construction.
- Refactored OPC Manager tests to utilize asynchronous mocks for improved accuracy in testing.
2025-11-10 09:14:35 -03:00
373245d5a1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics immediately after connection attempt

- Moved the emission of OPC_CONNECTIONS_TOTAL metric to occur right after the connection attempt, ensuring accurate tracking of connection status.
- Removed redundant metric emission logic that was previously placed after a successful connection.
2025-11-07 09:56:48 -03:00
032be28eb1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics after successful connection establishment

- Moved the metric emission for OPC_CONNECTIONS_TOTAL to occur after a successful connection.
- Removed redundant metric increment logic to streamline the connection process.
2025-11-07 08:58:11 -03:00
1cb687aa37 SIENTIAPDE-1325
Update metrics.py to reorganize OPC Manager metrics section

- Added a comment header for OPC Manager metrics for better code organization.
- Removed redundant comment header to streamline the code structure.
2025-11-07 08:49:09 -03:00
c4f8fe95a8 SIENTIAPDE-1325
Update IngestorManager to use asynchronous methods for checking OPC server cycles and listening status
2025-11-06 16:40:45 -03:00
254be06004 SIENTIAPDE-1325
Update requirements.txt to use version 1.5.3 of sientia-dataops-library and enhance DataManager and ResourceManager with improved logging and timestamping for data insertion and retrieval.
2025-11-05 12:38:35 -03:00
d93c497914 SIENTIAPDE-1325
Enhance Ingestor Class with Asynchronous Metrics and Monitoring Integration

- Refactored Ingestor class to inherit from SientiaMonitoring for improved observability.
- Updated methods to utilize asynchronous operations for declaring active ingestors and managing slots.
- Integrated pod_id label into various metrics for better tracking.
- Adjusted unit tests to reflect changes in the Ingestor initialization and asynchronous behavior.
2025-11-03 10:33:05 -03:00
dfd08e883f SIENTIAPDE-1325
Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations

- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
2025-11-03 09:41:49 -03:00
85a371a38b SIENTIAPDE-1325
SIENTIAPDE-1325: Refactor Ingestor and Manager Classes for Enhanced Asynchronous Operations

- Introduced asynchronous methods across Ingestor, IngestorManager, DataManager, and OpcManager to improve performance and responsiveness.
- Integrated MetricsController into various classes for better observability and monitoring.
- Updated Redis and MongoDB interactions to support asynchronous operations, enhancing data handling efficiency.
- Removed deprecated Redis metrics and streamlined resource management logic.
- Adjusted unit tests to accommodate the new asynchronous behavior and ensure proper mocking of async methods.
2025-10-31 16:32:45 -03:00
8e3d62ff76 SIENTIAPDE-1314
Refactor IngestorManager to streamline server disconnection process

- Simplified the logic in the `update_opc_servers` method by directly appending servers to the disconnection list and utilizing the new `remove_server` method for handling server shutdowns.
- Removed redundant code related to managing the `managed_tags` during server disconnection, enhancing code clarity and maintainability.
2025-10-23 15:17:47 -03:00
56023dfdf4 SIENTIAPDE-1314
Refactor IngestorManager and OpcManager for improved server management

- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
2025-10-23 15:11:32 -03:00
83e43edf95 Fix formatting of NOSONAR comment in update_opc_servers method for consistency 2025-10-23 11:58:00 -03:00
f415207d5b Update OPC server update method to include NOSONAR comment for code quality tools 2025-10-23 11:55:55 -03:00
eea6ccb48f Refactor OPC manager shutdown logic for improved reliability
Updated the IngestorManager class to ensure that OPC managers are properly shut down and removed from the management list when servers are lost or removed. This change enhances the integrity checks for OPC server management by ensuring asynchronous handling of shutdown operations.
2025-10-23 11:52:00 -03:00
6d2a0e46be Refactor OPC server integrity checks to be asynchronous
Updated the Ingestor and IngestorManager classes to use async methods for checking OPC server integrity and shutting down OPC managers. Adjusted related tests to ensure proper mocking of asynchronous behavior.
2025-10-23 11:46:02 -03:00
95c9ad85f9 SIENTIAPDE-1312
Update image tag in values.yaml and refactor OpcManager error handling

- Updated the image tag in values.yaml from "0.4.5" to "0.4.9" for the latest version.
- Refactored error handling in OpcManager to improve logging and maintain code clarity.
- Adjusted unit tests for better readability and consistency in assertions.
2025-10-21 17:03:58 -03:00
3eacf27faf SIENTIAPDE-1312
Refactor OpcManager connection error handling and enhance unit tests

- Simplified error handling during connection attempts in OpcManager by removing redundant disconnection logic.
- Updated unit tests to include subscription_period_ms in server configuration and adjusted connection assertions to include a timeout parameter.
- Added new tests for disconnection fallback functionality to ensure robust error handling during disconnect attempts.
2025-10-20 17:08:40 -03:00
aaa8d00f56 SIENTIAPDE-1312
Update OpcManager to set a timeout for the client connection

- Added a timeout parameter to the Client initialization in OpcManager to enhance connection reliability and prevent indefinite blocking during connection attempts.
2025-10-20 16:26:24 -03:00
8c552f50a8 SIENTIAPDE-1312
Fix pod URI formatting and enhance disconnection error logging in OpcManager

- Updated pod URI construction to replace hyphens with colons for correct formatting.
- Improved error logging during disconnection attempts to include the attempt number for better traceability.
- Changed disconnection_fallback method call to be awaited for proper asynchronous handling.
2025-10-20 15:56:00 -03:00
7001b5e9bd SIENTIAPDE-1312
SIENTIAPDE-1318 Enhance OpcManager disconnection handling

- Introduced a disconnection_fallback method to attempt multiple disconnects from the OPC UA server, improving reliability.
- Updated error handling during connection attempts to ensure proper logging and notification of disconnection failures.
- Refactored disconnect method to utilize the new fallback mechanism and maintain metrics accurately.
2025-10-20 15:27:05 -03:00