gitea_admin/sientia-dataops-scouter_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
91 Commits 3 Branches 0 Tags
879c1fdc495939afe04db1d776a21790245fee77
Commit Graph

91 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vitor-aignosi
879c1fdc49 SIENTIAPDE-1110 
Add logging for data loading in Druid activity, enhancing traceability of data retrieval intervals.
 2025-07-02 09:27:02 -03:00
vitor-aignosi
c3c8e1b766 SIENTIAPDE-1110 
Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
 2025-07-02 09:25:46 -03:00
vitor-aignosi
332612ac84 SIENTIAPDE-1110 
Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
 2025-07-02 09:02:59 -03:00
vitor-aignosi
91f2d8610b SIENTIAPDE-1110 
Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
 2025-07-01 16:09:46 -03:00
vitor-aignosi
defac7b1f0 SIENTIAPDE-1110 
Update Redis activity to include a specific prefix for the last data timestamp key, enhancing clarity in key management.
 2025-07-01 16:06:52 -03:00
vitor-aignosi
82ce1852e3 SIENTIAPDE-1110 
Add debug logging for data filter in MongoDB activity to enhance traceability of filtering criteria.
 2025-07-01 16:03:40 -03:00
vitor-aignosi
b718801290 SIENTIAPDE-1110 
Update get_last_data_timestamp method in Redis activity to return a string or None, improving type clarity for timestamp retrieval.
 2025-07-01 15:53:47 -03:00
vitor-aignosi
337a54217c SIENTIAPDE-1110 
Fix aggregation function key in Gates activity from 'aggr_function' to 'aggr_func' for consistency in data processing.
 2025-07-01 15:48:18 -03:00
vitor-aignosi
0e915f5491 SIENTIAPDE-1110 
Update Redis activity to include specific prefixes for keys in last data timestamp and held data, enhancing clarity in data management.
 2025-07-01 15:45:03 -03:00
vitor-aignosi
c9b9974f9d SIENTIAPDE-1110 
Refactor Redis activity to remove unused 'ttl' parameter in set method, streamlining data handling.
 2025-07-01 15:41:49 -03:00
vitor-aignosi
f183bac90d SIENTIAPDE-1110 
Enhance MongoDB data return format by converting to DataFrame dictionary, improving data structure for downstream processing.
 2025-07-01 15:37:40 -03:00
vitor-aignosi
f441873969 SIENTIAPDE-1110 
Refactor MongoDB data retrieval to convert cursor to list, improving data handling and compatibility.
 2025-07-01 15:33:51 -03:00
vitor-aignosi
d7d44556c8 SIENTIAPDE-1110 
Update MongoDB activity to handle None last_data_timestamp by using an empty filter, improving data retrieval logic.
 2025-07-01 15:31:27 -03:00
vitor-aignosi
1908741feb SIENTIAPDE-1110 
Add debug logging for MongoDB data loading to enhance traceability of input data.
 2025-07-01 15:26:57 -03:00
vitor-aignosi
9b524ea428 SIENTIAPDE-1110 
Refactor MongoDB activity to import datetime directly and format 'inserted_at' timestamps to ISO 8601 format for improved data consistency.
 2025-07-01 15:20:50 -03:00
vitor-aignosi
51f4eb1435 SIENTIAPDE-1110 
Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
 2025-07-01 15:16:55 -03:00
vitor-aignosi
97d66ed6cd SIENTIAPDE-1110 
Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
 2025-07-01 15:15:18 -03:00
vitor-aignosi
53337de74a SIENTIAPDE-1110 
SIENTIAPDE-1110 Add MongoDB integration and enhance Redis activity with timestamp management functions.
 2025-07-01 15:10:32 -03:00
vitor-aignosi
e7b05ebf3a SIENTIAPDE-1110 
Add logging for consumer creation in Kafka activity to enhance traceability of topic management.
 2025-07-01 11:05:03 -03:00
vitor-aignosi
946d953598 SIENTIAPDE-1110 
Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
 2025-07-01 10:57:12 -03:00
vitor-aignosi
704099c387 SIENTIAPDE-1110 
Update Kafka activity to use AIOKafkaConsumer for asynchronous message handling, enhancing consumer management and adding support for multiple topics.
 2025-07-01 10:44:37 -03:00
vitor-aignosi
66b568838e SIENTIAPDE-1110 
Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
 2025-07-01 08:21:00 -03:00
vitor-aignosi
a53804331b SIENTIAPDE-1110 
Enhance Kafka activity initialization by adding bootstrap_servers and group_id attributes for improved configuration management.
 2025-06-30 14:36:15 -03:00
vitor-aignosi
d33a5cbb72 SIENTIAPDE-1110 
Refactor Kafka activity to use local kafka_connector variable for improved readability and ensure proper closure of the Kafka consumer after polling.
 2025-06-30 14:12:02 -03:00
vitor-aignosi
6144bcf8ff SIENTIAPDE-1110 
Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
 2025-06-30 14:11:30 -03:00
vitor-aignosi
9afb74be21 SIENTIAPDE-1110 
Add metadata to input_data in Scouter and update CoreScouter to use it
 2025-06-27 10:07:43 -03:00
vitor-aignosi
74a2d9aa3e SIENTIAPDE-1110 
Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
 2025-06-27 09:34:58 -03:00
vitor-aignosi
99d80020d0 Merge pull request #6 from Aignosi/SIENTIAPDE-1110-criar-testes-e-2-e 
Sientiapde 1110 criar testes e 2 e
 2025-06-26 16:15:25 -03:00
vitor-aignosi
f6831ff0aa SIENTIAPDE-1110 
Update image tag in values.yaml from 0.2.1 to 0.2.2
 2025-06-26 15:46:07 -03:00
vitor-aignosi
4a90b77786 SIENTIAPDE-1110 
Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
 2025-06-26 15:15:31 -03:00
vitor-aignosi
9355af1709 SIENTIAPDE-1110 
SIENTIAPDE-1110 Update image tag in values.yaml from 0.1.0-uat to 0.4.0-uat
 2025-06-25 10:07:54 -03:00
vitor-aignosi
92ddaaf17a SIENTIAPDE-1110 
Update image tag in values.yaml from 0.0.2 to 0.2.1
 2025-06-25 09:54:17 -03:00
Bruno Domingues
d31812c7fc SIENTIAPDE-1111: refactor sonarqube pipeline 2. 2025-06-23 22:04:27 -03:00
Bruno Domingues
cff067c517 SIENTIAPDE-1111: refactor sonarqube pipeline. 2025-06-23 21:59:08 -03:00
Matheus Demoner
b6f3e4f813 Merge pull request #5 from Aignosi/SIENTIAPDE-1100-rever-erros-nao-lancados-lancando-os-se-necessario 
SIENTIAPDE-1100
 2025-06-12 10:42:40 -03:00
vitor-aignosi
d27dde99a9 SIENTIAPDE-1100 
Fix error handling in aggregate_data method by raising the caught exception
 2025-06-12 09:27:37 -03:00
Matheus Demoner
7b2a5260b3 Merge pull request #4 from Aignosi/SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious 
SIENTIAPDE-1082
 2025-06-09 16:44:57 -03:00
vitor-aignosi
577513576d Merge branch 'main' into SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious 2025-06-09 15:14:00 -03:00
vitor-aignosi
f541e05e29 Update sientia-mlops-library version to 0.38.1 in requirements.txt 2025-06-09 15:12:43 -03:00
vitor-aignosi
8cf3273714 SIENTIAPDE-1082 
Implement shutdown methods for Activities and Kafka; enhance error handling in worker main function
 2025-06-09 15:09:53 -03:00
Bruno Domingues
ab0c21f5fb Merge pull request #3 from Aignosi/SIENTIAPDE-1094-atualizar-versao-da-lib 
Sientiapde 1094 atualizar versao da lib
 2025-06-09 13:33:51 -03:00
vitor-aignosi
b8beb0e151 SIENTIAPDE-1094 
Refactor Redis activity tests to improve initialization and assertion logic
 2025-06-09 11:06:53 -03:00
vitor-aignosi
3759f10aec SIENTIAPDE-1094 
Refactor test imports and remove unused test files
 2025-06-09 10:37:06 -03:00
vitor-aignosi
7dbb9a29ea SIENTIAPDE-1094 
Update environment configuration and refactor activity imports

- Changed Kafka, Redis, and Temporal host configurations to use localhost.
- Updated the version reference for the sientia-dataops-library in requirements.txt.
- Refactored import paths for activities to align with new module structure.
- Removed unused base.py and postgres.py files.
- Updated logger and policies imports to reflect new module locations.
- Adjusted values.yaml for branch and log level settings.
 2025-06-09 09:43:06 -03:00
vitor-aignosi
deafb335a1 Merge pull request #2 from Aignosi/SIENTIAPDE-1005-implementar-os-workflows-mapeados-utilizando-as-workers-e-activities-apropriadas 
Sientiapde 1005 implementar os workflows mapeados utilizando as workers e activities apropriadas
 2025-05-20 14:37:24 -03:00
vitor-aignosi
7822a41040 Adjusting actions 2025-05-19 16:51:22 -03:00
vitor-aignosi
451dc5fbb7 Removing coveragerc 2025-05-19 16:42:50 -03:00
vitor-aignosi
1839240495 core scouter fix 2025-05-19 16:29:45 -03:00
vitor-aignosi
94aa989029 actions version 2025-05-19 16:24:49 -03:00
vitor-aignosi
33a852c013 Merge branch 'main' into SIENTIAPDE-1005-implementar-os-workflows-mapeados-utilizando-as-workers-e-activities-apropriadas 2025-05-19 16:10:43 -03:00