430fe04202
SIENTIAPDE-1110
...
Enhance Druid activity by adding configuration for Druid host and port in values.yaml. Refactor data loading logic to utilize SQLAlchemy for querying Druid, improving query efficiency and readability. Update timestamp handling to streamline data retrieval process.
2025-07-02 10:19:04 -03:00
5decb018d2
SIENTIAPDE-1110
2025-07-02 09:27:14 -03:00
879c1fdc49
SIENTIAPDE-1110
...
Add logging for data loading in Druid activity, enhancing traceability of data retrieval intervals.
2025-07-02 09:27:02 -03:00
c3c8e1b766
SIENTIAPDE-1110
...
Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
2025-07-02 09:25:46 -03:00
332612ac84
SIENTIAPDE-1110
...
Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
2025-07-02 09:02:59 -03:00
91f2d8610b
SIENTIAPDE-1110
...
Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
2025-07-01 16:09:46 -03:00
defac7b1f0
SIENTIAPDE-1110
...
Update Redis activity to include a specific prefix for the last data timestamp key, enhancing clarity in key management.
2025-07-01 16:06:52 -03:00
82ce1852e3
SIENTIAPDE-1110
...
Add debug logging for data filter in MongoDB activity to enhance traceability of filtering criteria.
2025-07-01 16:03:40 -03:00
b718801290
SIENTIAPDE-1110
...
Update get_last_data_timestamp method in Redis activity to return a string or None, improving type clarity for timestamp retrieval.
2025-07-01 15:53:47 -03:00
337a54217c
SIENTIAPDE-1110
...
Fix aggregation function key in Gates activity from 'aggr_function' to 'aggr_func' for consistency in data processing.
2025-07-01 15:48:18 -03:00
0e915f5491
SIENTIAPDE-1110
...
Update Redis activity to include specific prefixes for keys in last data timestamp and held data, enhancing clarity in data management.
2025-07-01 15:45:03 -03:00
c9b9974f9d
SIENTIAPDE-1110
...
Refactor Redis activity to remove unused 'ttl' parameter in set method, streamlining data handling.
2025-07-01 15:41:49 -03:00
f183bac90d
SIENTIAPDE-1110
...
Enhance MongoDB data return format by converting to DataFrame dictionary, improving data structure for downstream processing.
2025-07-01 15:37:40 -03:00
f441873969
SIENTIAPDE-1110
...
Refactor MongoDB data retrieval to convert cursor to list, improving data handling and compatibility.
2025-07-01 15:33:51 -03:00
d7d44556c8
SIENTIAPDE-1110
...
Update MongoDB activity to handle None last_data_timestamp by using an empty filter, improving data retrieval logic.
2025-07-01 15:31:27 -03:00
1908741feb
SIENTIAPDE-1110
...
Add debug logging for MongoDB data loading to enhance traceability of input data.
2025-07-01 15:26:57 -03:00
9b524ea428
SIENTIAPDE-1110
...
Refactor MongoDB activity to import datetime directly and format 'inserted_at' timestamps to ISO 8601 format for improved data consistency.
2025-07-01 15:20:50 -03:00
51f4eb1435
SIENTIAPDE-1110
...
Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
2025-07-01 15:16:55 -03:00
97d66ed6cd
SIENTIAPDE-1110
...
Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
2025-07-01 15:15:18 -03:00
53337de74a
SIENTIAPDE-1110
...
SIENTIAPDE-1110 Add MongoDB integration and enhance Redis activity with timestamp management functions.
2025-07-01 15:10:32 -03:00
e7b05ebf3a
SIENTIAPDE-1110
...
Add logging for consumer creation in Kafka activity to enhance traceability of topic management.
2025-07-01 11:05:03 -03:00
946d953598
SIENTIAPDE-1110
...
Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
2025-07-01 10:57:12 -03:00
704099c387
SIENTIAPDE-1110
...
Update Kafka activity to use AIOKafkaConsumer for asynchronous message handling, enhancing consumer management and adding support for multiple topics.
2025-07-01 10:44:37 -03:00
66b568838e
SIENTIAPDE-1110
...
Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
2025-07-01 08:21:00 -03:00
a53804331b
SIENTIAPDE-1110
...
Enhance Kafka activity initialization by adding bootstrap_servers and group_id attributes for improved configuration management.
2025-06-30 14:36:15 -03:00
d33a5cbb72
SIENTIAPDE-1110
...
Refactor Kafka activity to use local kafka_connector variable for improved readability and ensure proper closure of the Kafka consumer after polling.
2025-06-30 14:12:02 -03:00
6144bcf8ff
SIENTIAPDE-1110
...
Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
2025-06-30 14:11:30 -03:00
9afb74be21
SIENTIAPDE-1110
...
Add metadata to input_data in Scouter and update CoreScouter to use it
2025-06-27 10:07:43 -03:00
74a2d9aa3e
SIENTIAPDE-1110
...
Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
2025-06-27 09:34:58 -03:00
99d80020d0
Merge pull request #6 from Aignosi/SIENTIAPDE-1110-criar-testes-e-2-e
...
Sientiapde 1110 criar testes e 2 e
2025-06-26 16:15:25 -03:00
f6831ff0aa
SIENTIAPDE-1110
...
Update image tag in values.yaml from 0.2.1 to 0.2.2
2025-06-26 15:46:07 -03:00
4a90b77786
SIENTIAPDE-1110
...
Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
2025-06-26 15:15:31 -03:00
9355af1709
SIENTIAPDE-1110
...
SIENTIAPDE-1110 Update image tag in values.yaml from 0.1.0-uat to 0.4.0-uat
2025-06-25 10:07:54 -03:00
92ddaaf17a
SIENTIAPDE-1110
...
Update image tag in values.yaml from 0.0.2 to 0.2.1
2025-06-25 09:54:17 -03:00
d31812c7fc
SIENTIAPDE-1111: refactor sonarqube pipeline 2.
2025-06-23 22:04:27 -03:00
cff067c517
SIENTIAPDE-1111: refactor sonarqube pipeline.
2025-06-23 21:59:08 -03:00
Matheus Demoner
b6f3e4f813
Merge pull request #5 from Aignosi/SIENTIAPDE-1100-rever-erros-nao-lancados-lancando-os-se-necessario
...
SIENTIAPDE-1100
2025-06-12 10:42:40 -03:00
d27dde99a9
SIENTIAPDE-1100
...
Fix error handling in aggregate_data method by raising the caught exception
2025-06-12 09:27:37 -03:00
Matheus Demoner
7b2a5260b3
Merge pull request #4 from Aignosi/SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious
...
SIENTIAPDE-1082
2025-06-09 16:44:57 -03:00
577513576d
Merge branch 'main' into SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious
2025-06-09 15:14:00 -03:00
f541e05e29
Update sientia-mlops-library version to 0.38.1 in requirements.txt
2025-06-09 15:12:43 -03:00
8cf3273714
SIENTIAPDE-1082
...
Implement shutdown methods for Activities and Kafka; enhance error handling in worker main function
2025-06-09 15:09:53 -03:00
Bruno Domingues
ab0c21f5fb
Merge pull request #3 from Aignosi/SIENTIAPDE-1094-atualizar-versao-da-lib
...
Sientiapde 1094 atualizar versao da lib
2025-06-09 13:33:51 -03:00
b8beb0e151
SIENTIAPDE-1094
...
Refactor Redis activity tests to improve initialization and assertion logic
2025-06-09 11:06:53 -03:00
3759f10aec
SIENTIAPDE-1094
...
Refactor test imports and remove unused test files
2025-06-09 10:37:06 -03:00
7dbb9a29ea
SIENTIAPDE-1094
...
Update environment configuration and refactor activity imports
- Changed Kafka, Redis, and Temporal host configurations to use localhost.
- Updated the version reference for the sientia-dataops-library in requirements.txt.
- Refactored import paths for activities to align with new module structure.
- Removed unused base.py and postgres.py files.
- Updated logger and policies imports to reflect new module locations.
- Adjusted values.yaml for branch and log level settings.
2025-06-09 09:43:06 -03:00
deafb335a1
Merge pull request #2 from Aignosi/SIENTIAPDE-1005-implementar-os-workflows-mapeados-utilizando-as-workers-e-activities-apropriadas
...
Sientiapde 1005 implementar os workflows mapeados utilizando as workers e activities apropriadas
2025-05-20 14:37:24 -03:00
7822a41040
Adjusting actions
2025-05-19 16:51:22 -03:00
451dc5fbb7
Removing coveragerc
2025-05-19 16:42:50 -03:00
1839240495
core scouter fix
2025-05-19 16:29:45 -03:00
