Commit Graph

285 Commits

Author SHA1 Message Date
PedroHMCosme
905da5151b feat(gateway): DataSink.update_write_status Postgres by timestamp+model_id (C3); drop unused metadata param 2026-07-14 19:24:24 -03:00
PedroHMCosme
86d0f6cf23 fix(gateway): make WriteEngine + converter raise-safe against malformed write commands 2026-07-14 19:18:52 -03:00
PedroHMCosme
dcf8369638 feat(gateway): WriteEngine (status_code/timestamp/flags, safe converter, rate-limit) 2026-07-14 16:59:43 -03:00
PedroHMCosme
2a9af05464 SIENTIAPDE-1972: F1 close-out - ledger de commits + carry-forward notes (Task 10.end) 2026-07-14 16:33:42 -03:00
PedroHMCosme
00c74e5606 feat(gateway): SubscriptionEngine multi-name fan-out (C1) 2026-07-14 16:17:13 -03:00
PedroHMCosme
323bf24a0f refactor(gateway): remove legacy Kafka egress (MongoDB-only) 2026-07-14 15:58:55 -03:00
PedroHMCosme
ae88ac662e feat(gateway): DataSink.publish_reading multi-name fan-out 2026-07-14 10:51:52 -03:00
PedroHMCosme
44df022e32 feat(gateway): slot v2 loader (contract C1) 2026-07-14 10:47:33 -03:00
PedroHMCosme
00f919cec6 SIENTIAPDE-1972: plano - modelo de PR por fase + ledger de commits (F0 registrado) + close-out por fase 2026-07-14 10:37:09 -03:00
PedroHMCosme
b57be9f5fe refactor(gateway): drop dead error_stack and unused metadata from OpcConnectionPool (ponytail) 2026-07-14 10:06:59 -03:00
PedroHMCosme
0b29857089 fix(gateway): per-server lock in OpcConnectionPool to avoid duplicate connects 2026-07-14 09:50:14 -03:00
PedroHMCosme
a08fd88c80 test(ingestor_manager): force gc.collect() after each test to avoid __del__ flake
IngestorManager.__del__ calls asyncio.run(self.shutdown()). Adding
tests/unit/connection/test_pool.py shifted GC timing enough that a
cyclic-referenced IngestorManager instance from an earlier test was
swept mid test___del__, double-calling the patched asyncio mock and
failing the assertion. Deterministic gc.collect() after each test in
this file keeps finalizers scoped to their own test.
2026-07-14 09:36:01 -03:00
PedroHMCosme
5d30a5c547 feat(gateway): OpcConnectionPool (per-server, reuse, idle-evict) 2026-07-14 09:35:53 -03:00
PedroHMCosme
09e42901d1 SIENTIAPDE-1972: remover egress Kafka do Gateway (Mongo-only) no spec/plano 2026-07-14 09:06:53 -03:00
PedroHMCosme
bbe10a4f73 SIENTIAPDE-1972: resolve chamador do GRPC (sem consumidor hoje; interno, sem auth) 2026-07-14 08:51:21 -03:00
PedroHMCosme
ec2a324a84 feat(gateway): cert provider from MinIO+Mongo (sientia-dataops) 2026-07-13 19:20:41 -03:00
PedroHMCosme
b05c2fd76a feat(gateway): Basic256Sha256 + channel>session lifetime (1879) 2026-07-13 18:47:33 -03:00
PedroHMCosme
d6cb1e492d feat(gateway): monotonic clock helper (1879) 2026-07-13 18:39:23 -03:00
PedroHMCosme
55f67216ee feat(gateway): safe type converter (1879 BUG-1/BUG-2) 2026-07-13 18:31:07 -03:00
PedroHMCosme
3dd4f60b0e feat(gateway): add write/active-read/error metrics 2026-07-13 18:27:30 -03:00
PedroHMCosme
0694f6ac87 chore(gateway): add nats-py, grpcio deps (Task 1) 2026-07-13 18:24:32 -03:00
PedroHMCosme
dd41145139 SIENTIAPDE-1972: plano de implementacao do OPC Gateway (F0-F4, TDD) + revisao 2026-07-13 18:12:08 -03:00
PedroHMCosme
4225789dc8 SIENTIAPDE-1972: localiza criacao das tabelas de predicao (pandas to_sql), adiciona schema/table_name ao contrato NATS e fixa cert bucket aprovado 2026-07-13 17:46:43 -03:00
PedroHMCosme
7cffc2629d SIENTIAPDE-1972: confirma criacao dinamica das tabelas de predicao (coluna no template + backfill) 2026-07-13 17:25:27 -03:00
PedroHMCosme
c31db70c8a SIENTIAPDE-1972: fixa decisoes (JOIN=tag_id, certs=sientia-dataops, metadata vazio) e coluna de write_status via Flyway em tabelas por-modelo 2026-07-13 17:17:19 -03:00
PedroHMCosme
941c6c0029 SIENTIAPDE-1972: resolve bucket de certs (sientia-dataops), mecanica do JOIN 1974 e discrepancia do campo metadata (1600) 2026-07-13 17:09:07 -03:00
PedroHMCosme
8a05ff70da SIENTIAPDE-1972: resolve stack de migracao (Flyway) no spec 2026-07-13 16:58:54 -03:00
PedroHMCosme
524f4d6d3f SIENTIAPDE-1972: fixa NATS como broker decidido (brief/Vitor) e reflete rename da branch 2026-07-13 16:55:27 -03:00
PedroHMCosme
c980092d95 SIENTIAPDE-1972: enriquece design do OPC Gateway com padroes de plataforma (aig-docs) e corrige contrato de status de escrita 2026-07-13 16:47:59 -03:00
PedroHMCosme
6f6f3bf734 SIENTIAPDE-1972: design doc do OPC Gateway (arquitetura, contratos e faseamento) 2026-07-13 16:29:57 -03:00
Bruno Domingues
498030dd5f Merge pull request #28 from Aignosi/feature/SIENTIAPDE-1945
SIENTIAPDE-1945: Remove values.yaml
2026-07-10 11:50:21 -03:00
878644c953 SIENTIAPDE-1945: Remove values.yaml. 2026-07-08 22:12:59 -03:00
1e0c83828e chore(sonar): Update project key 2026-07-01 21:11:01 -03:00
b74ec5a08e Merge pull request #27 from Aignosi/feature/SIENTIAPDE-1325-adicionar-metricas-especificas-de-operacoes-externas
SIENTIAPDE-1325: Enhance Asynchronous Operations, Metrics, and Release Workflow
2025-11-10 10:12:05 -03:00
89278827e7 SIENTIAPDE-1325
Enhance release workflow to trigger only on merged pull requests and streamline data insertion in DataManager

- Updated the release workflow to execute only when a pull request is merged.
- Simplified the document insertion logic in DataManager by consolidating the dictionary construction.
- Refactored OPC Manager tests to utilize asynchronous mocks for improved accuracy in testing.
2025-11-10 09:14:35 -03:00
373245d5a1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics immediately after connection attempt

- Moved the emission of OPC_CONNECTIONS_TOTAL metric to occur right after the connection attempt, ensuring accurate tracking of connection status.
- Removed redundant metric emission logic that was previously placed after a successful connection.
2025-11-07 09:56:48 -03:00
032be28eb1 SIENTIAPDE-1325
Refactor OpcManager to emit connection metrics after successful connection establishment

- Moved the metric emission for OPC_CONNECTIONS_TOTAL to occur after a successful connection.
- Removed redundant metric increment logic to streamline the connection process.
2025-11-07 08:58:11 -03:00
1cb687aa37 SIENTIAPDE-1325
Update metrics.py to reorganize OPC Manager metrics section

- Added a comment header for OPC Manager metrics for better code organization.
- Removed redundant comment header to streamline the code structure.
2025-11-07 08:49:09 -03:00
c4f8fe95a8 SIENTIAPDE-1325
Update IngestorManager to use asynchronous methods for checking OPC server cycles and listening status
2025-11-06 16:40:45 -03:00
254be06004 SIENTIAPDE-1325
Update requirements.txt to use version 1.5.3 of sientia-dataops-library and enhance DataManager and ResourceManager with improved logging and timestamping for data insertion and retrieval.
2025-11-05 12:38:35 -03:00
e313de453e SIENTIAPDE-1325
Update requirements.txt to use version 1.5.2 of sientia-dataops-library and bump image tag in values.yaml to 0.5.0 for consistency with recent changes.
2025-11-05 09:15:36 -03:00
d93c497914 SIENTIAPDE-1325
Enhance Ingestor Class with Asynchronous Metrics and Monitoring Integration

- Refactored Ingestor class to inherit from SientiaMonitoring for improved observability.
- Updated methods to utilize asynchronous operations for declaring active ingestors and managing slots.
- Integrated pod_id label into various metrics for better tracking.
- Adjusted unit tests to reflect changes in the Ingestor initialization and asynchronous behavior.
2025-11-03 10:33:05 -03:00
dfd08e883f SIENTIAPDE-1325
Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations

- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
2025-11-03 09:41:49 -03:00
85a371a38b SIENTIAPDE-1325
SIENTIAPDE-1325: Refactor Ingestor and Manager Classes for Enhanced Asynchronous Operations

- Introduced asynchronous methods across Ingestor, IngestorManager, DataManager, and OpcManager to improve performance and responsiveness.
- Integrated MetricsController into various classes for better observability and monitoring.
- Updated Redis and MongoDB interactions to support asynchronous operations, enhancing data handling efficiency.
- Removed deprecated Redis metrics and streamlined resource management logic.
- Adjusted unit tests to accommodate the new asynchronous behavior and ensure proper mocking of async methods.
2025-10-31 16:32:45 -03:00
4158a74cac SIENTIAPDE-1325
Update requirements.txt and values.yaml to reflect new library path and GITHUB_BRANCH for feature SIENTIAPDE-1325
2025-10-30 08:43:32 -03:00
Matheus Demoner
f8e9d9679d Merge pull request #26 from Aignosi/fix/SIENTIAPDE-1314-ajustes-nas-camadas-de-monitoramento-do-sientia
SIENTIAPDE-1314: Refactor IngestorManager and OpcManager for Improved Server Management
2025-10-27 14:27:56 -03:00
92fd2cb356 SIENTIAPDE-1314
Update GITHUB_BRANCH in values.yaml to reflect new monitoring layer adjustments for SIENTIAPDE-1314
2025-10-24 10:14:13 -03:00
b1287acc1c SIENTIAPDE-1314
Enhance type hinting in encrypt.py for better code clarity

- Added type hint for the `file_tree` variable as a dictionary with string keys and Any values.
- Included a type ignore comment for the yaml import to address type checking issues.
2025-10-23 16:12:48 -03:00
8e3d62ff76 SIENTIAPDE-1314
Refactor IngestorManager to streamline server disconnection process

- Simplified the logic in the `update_opc_servers` method by directly appending servers to the disconnection list and utilizing the new `remove_server` method for handling server shutdowns.
- Removed redundant code related to managing the `managed_tags` during server disconnection, enhancing code clarity and maintainability.
2025-10-23 15:17:47 -03:00
56023dfdf4 SIENTIAPDE-1314
Refactor IngestorManager and OpcManager for improved server management

- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
2025-10-23 15:11:32 -03:00