vitor-aignosi
1dcba6b48e
SIENTIAPDE-1110
...
Refactor Scouter workflow to update data loading method, changing from load_latest_druid_data to load_latest_data. Adjusted parameters for consistency in collection naming and improved clarity in data retrieval process.
2025-07-03 14:04:10 -03:00
vitor-aignosi
3ee7e3687a
SIENTIAPDE-1110
...
Update test.ipynb to generate and save multiple pipeline specifications and input samples in JSON format. Modify values.yaml to increase replica count from 1 to 3 and adjust PostgreSQL max connections from 20 to 40 for improved scalability.
2025-07-03 14:00:09 -03:00
vitor-aignosi
208491498f
SIENTIAPDE-1110
...
Update Druid configuration in values.yaml to change port from 8081 to 8888. Refactor pydruid.py to use druid_engine for SQLAlchemy connections, enhancing clarity and consistency in data loading operations.
2025-07-02 10:35:53 -03:00
vitor-aignosi
6caec63977
SIENTIAPDE-1110
...
Refactor last data timestamp handling in Druid activity to use string format for default value, improving consistency in date representation.
2025-07-02 10:24:35 -03:00
vitor-aignosi
3bbc6a6a79
SIENTIAPDE-1110
...
Update Druid activity to handle None values for last data timestamp, ensuring default timestamp is set to epoch start. Enhance data loading debug logging for improved traceability.
2025-07-02 10:21:55 -03:00
vitor-aignosi
430fe04202
SIENTIAPDE-1110
...
Enhance Druid activity by adding configuration for Druid host and port in values.yaml. Refactor data loading logic to utilize SQLAlchemy for querying Druid, improving query efficiency and readability. Update timestamp handling to streamline data retrieval process.
2025-07-02 10:19:04 -03:00
vitor-aignosi
5decb018d2
SIENTIAPDE-1110
2025-07-02 09:27:14 -03:00
vitor-aignosi
879c1fdc49
SIENTIAPDE-1110
...
Add logging for data loading in Druid activity, enhancing traceability of data retrieval intervals.
2025-07-02 09:27:02 -03:00
vitor-aignosi
c3c8e1b766
SIENTIAPDE-1110
...
Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
2025-07-02 09:25:46 -03:00
vitor-aignosi
332612ac84
SIENTIAPDE-1110
...
Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
2025-07-02 09:02:59 -03:00
vitor-aignosi
91f2d8610b
SIENTIAPDE-1110
...
Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
2025-07-01 16:09:46 -03:00
vitor-aignosi
defac7b1f0
SIENTIAPDE-1110
...
Update Redis activity to include a specific prefix for the last data timestamp key, enhancing clarity in key management.
2025-07-01 16:06:52 -03:00
vitor-aignosi
82ce1852e3
SIENTIAPDE-1110
...
Add debug logging for data filter in MongoDB activity to enhance traceability of filtering criteria.
2025-07-01 16:03:40 -03:00
vitor-aignosi
b718801290
SIENTIAPDE-1110
...
Update get_last_data_timestamp method in Redis activity to return a string or None, improving type clarity for timestamp retrieval.
2025-07-01 15:53:47 -03:00
vitor-aignosi
337a54217c
SIENTIAPDE-1110
...
Fix aggregation function key in Gates activity from 'aggr_function' to 'aggr_func' for consistency in data processing.
2025-07-01 15:48:18 -03:00
vitor-aignosi
0e915f5491
SIENTIAPDE-1110
...
Update Redis activity to include specific prefixes for keys in last data timestamp and held data, enhancing clarity in data management.
2025-07-01 15:45:03 -03:00
vitor-aignosi
c9b9974f9d
SIENTIAPDE-1110
...
Refactor Redis activity to remove unused 'ttl' parameter in set method, streamlining data handling.
2025-07-01 15:41:49 -03:00
vitor-aignosi
f183bac90d
SIENTIAPDE-1110
...
Enhance MongoDB data return format by converting to DataFrame dictionary, improving data structure for downstream processing.
2025-07-01 15:37:40 -03:00
vitor-aignosi
f441873969
SIENTIAPDE-1110
...
Refactor MongoDB data retrieval to convert cursor to list, improving data handling and compatibility.
2025-07-01 15:33:51 -03:00
vitor-aignosi
d7d44556c8
SIENTIAPDE-1110
...
Update MongoDB activity to handle None last_data_timestamp by using an empty filter, improving data retrieval logic.
2025-07-01 15:31:27 -03:00
vitor-aignosi
1908741feb
SIENTIAPDE-1110
...
Add debug logging for MongoDB data loading to enhance traceability of input data.
2025-07-01 15:26:57 -03:00
vitor-aignosi
9b524ea428
SIENTIAPDE-1110
...
Refactor MongoDB activity to import datetime directly and format 'inserted_at' timestamps to ISO 8601 format for improved data consistency.
2025-07-01 15:20:50 -03:00
vitor-aignosi
51f4eb1435
SIENTIAPDE-1110
...
Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
2025-07-01 15:16:55 -03:00
vitor-aignosi
97d66ed6cd
SIENTIAPDE-1110
...
Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
2025-07-01 15:15:18 -03:00
vitor-aignosi
53337de74a
SIENTIAPDE-1110
...
SIENTIAPDE-1110 Add MongoDB integration and enhance Redis activity with timestamp management functions.
2025-07-01 15:10:32 -03:00
vitor-aignosi
e7b05ebf3a
SIENTIAPDE-1110
...
Add logging for consumer creation in Kafka activity to enhance traceability of topic management.
2025-07-01 11:05:03 -03:00
vitor-aignosi
946d953598
SIENTIAPDE-1110
...
Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
2025-07-01 10:57:12 -03:00
vitor-aignosi
704099c387
SIENTIAPDE-1110
...
Update Kafka activity to use AIOKafkaConsumer for asynchronous message handling, enhancing consumer management and adding support for multiple topics.
2025-07-01 10:44:37 -03:00
vitor-aignosi
66b568838e
SIENTIAPDE-1110
...
Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
2025-07-01 08:21:00 -03:00
vitor-aignosi
a53804331b
SIENTIAPDE-1110
...
Enhance Kafka activity initialization by adding bootstrap_servers and group_id attributes for improved configuration management.
2025-06-30 14:36:15 -03:00
vitor-aignosi
d33a5cbb72
SIENTIAPDE-1110
...
Refactor Kafka activity to use local kafka_connector variable for improved readability and ensure proper closure of the Kafka consumer after polling.
2025-06-30 14:12:02 -03:00
vitor-aignosi
6144bcf8ff
SIENTIAPDE-1110
...
Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
2025-06-30 14:11:30 -03:00
vitor-aignosi
9afb74be21
SIENTIAPDE-1110
...
Add metadata to input_data in Scouter and update CoreScouter to use it
2025-06-27 10:07:43 -03:00
vitor-aignosi
74a2d9aa3e
SIENTIAPDE-1110
...
Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
2025-06-27 09:34:58 -03:00
vitor-aignosi
99d80020d0
Merge pull request #6 from Aignosi/SIENTIAPDE-1110-criar-testes-e-2-e
...
Sientiapde 1110 criar testes e 2 e
2025-06-26 16:15:25 -03:00
vitor-aignosi
f6831ff0aa
SIENTIAPDE-1110
...
Update image tag in values.yaml from 0.2.1 to 0.2.2
2025-06-26 15:46:07 -03:00
vitor-aignosi
4a90b77786
SIENTIAPDE-1110
...
Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
2025-06-26 15:15:31 -03:00
vitor-aignosi
9355af1709
SIENTIAPDE-1110
...
SIENTIAPDE-1110 Update image tag in values.yaml from 0.1.0-uat to 0.4.0-uat
2025-06-25 10:07:54 -03:00
vitor-aignosi
92ddaaf17a
SIENTIAPDE-1110
...
Update image tag in values.yaml from 0.0.2 to 0.2.1
2025-06-25 09:54:17 -03:00
Bruno Domingues
d31812c7fc
SIENTIAPDE-1111: refactor sonarqube pipeline 2.
2025-06-23 22:04:27 -03:00
Bruno Domingues
cff067c517
SIENTIAPDE-1111: refactor sonarqube pipeline.
2025-06-23 21:59:08 -03:00
Matheus Demoner
b6f3e4f813
Merge pull request #5 from Aignosi/SIENTIAPDE-1100-rever-erros-nao-lancados-lancando-os-se-necessario
...
SIENTIAPDE-1100
2025-06-12 10:42:40 -03:00
vitor-aignosi
d27dde99a9
SIENTIAPDE-1100
...
Fix error handling in aggregate_data method by raising the caught exception
2025-06-12 09:27:37 -03:00
Matheus Demoner
7b2a5260b3
Merge pull request #4 from Aignosi/SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious
...
SIENTIAPDE-1082
2025-06-09 16:44:57 -03:00
vitor-aignosi
577513576d
Merge branch 'main' into SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious
2025-06-09 15:14:00 -03:00
vitor-aignosi
f541e05e29
Update sientia-mlops-library version to 0.38.1 in requirements.txt
2025-06-09 15:12:43 -03:00
vitor-aignosi
8cf3273714
SIENTIAPDE-1082
...
Implement shutdown methods for Activities and Kafka; enhance error handling in worker main function
2025-06-09 15:09:53 -03:00
Bruno Domingues
ab0c21f5fb
Merge pull request #3 from Aignosi/SIENTIAPDE-1094-atualizar-versao-da-lib
...
Sientiapde 1094 atualizar versao da lib
2025-06-09 13:33:51 -03:00
vitor-aignosi
b8beb0e151
SIENTIAPDE-1094
...
Refactor Redis activity tests to improve initialization and assertion logic
2025-06-09 11:06:53 -03:00
vitor-aignosi
3759f10aec
SIENTIAPDE-1094
...
Refactor test imports and remove unused test files
2025-06-09 10:37:06 -03:00
