gitea_admin/sientia-dataops-scouter_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
147 Commits 3 Branches 0 Tags
40c57ed23dd302129a8c1455b344a1f12f1df575
Commit Graph

147 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vitor-aignosi
3ee7e3687a SIENTIAPDE-1110 
Update test.ipynb to generate and save multiple pipeline specifications and input samples in JSON format. Modify values.yaml to increase replica count from 1 to 3 and adjust PostgreSQL max connections from 20 to 40 for improved scalability.
 2025-07-03 14:00:09 -03:00
vitor-aignosi
208491498f SIENTIAPDE-1110 
Update Druid configuration in values.yaml to change port from 8081 to 8888. Refactor pydruid.py to use druid_engine for SQLAlchemy connections, enhancing clarity and consistency in data loading operations.
 2025-07-02 10:35:53 -03:00
vitor-aignosi
6caec63977 SIENTIAPDE-1110 
Refactor last data timestamp handling in Druid activity to use string format for default value, improving consistency in date representation.
 2025-07-02 10:24:35 -03:00
vitor-aignosi
3bbc6a6a79 SIENTIAPDE-1110 
Update Druid activity to handle None values for last data timestamp, ensuring default timestamp is set to epoch start. Enhance data loading debug logging for improved traceability.
 2025-07-02 10:21:55 -03:00
vitor-aignosi
430fe04202 SIENTIAPDE-1110 
Enhance Druid activity by adding configuration for Druid host and port in values.yaml. Refactor data loading logic to utilize SQLAlchemy for querying Druid, improving query efficiency and readability. Update timestamp handling to streamline data retrieval process.
 2025-07-02 10:19:04 -03:00
vitor-aignosi
5decb018d2 SIENTIAPDE-1110 2025-07-02 09:27:14 -03:00
vitor-aignosi
879c1fdc49 SIENTIAPDE-1110 
Add logging for data loading in Druid activity, enhancing traceability of data retrieval intervals.
 2025-07-02 09:27:02 -03:00
vitor-aignosi
c3c8e1b766 SIENTIAPDE-1110 
Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
 2025-07-02 09:25:46 -03:00
vitor-aignosi
332612ac84 SIENTIAPDE-1110 
Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
 2025-07-02 09:02:59 -03:00
vitor-aignosi
91f2d8610b SIENTIAPDE-1110 
Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
 2025-07-01 16:09:46 -03:00
vitor-aignosi
defac7b1f0 SIENTIAPDE-1110 
Update Redis activity to include a specific prefix for the last data timestamp key, enhancing clarity in key management.
 2025-07-01 16:06:52 -03:00
vitor-aignosi
82ce1852e3 SIENTIAPDE-1110 
Add debug logging for data filter in MongoDB activity to enhance traceability of filtering criteria.
 2025-07-01 16:03:40 -03:00
vitor-aignosi
b718801290 SIENTIAPDE-1110 
Update get_last_data_timestamp method in Redis activity to return a string or None, improving type clarity for timestamp retrieval.
 2025-07-01 15:53:47 -03:00
vitor-aignosi
337a54217c SIENTIAPDE-1110 
Fix aggregation function key in Gates activity from 'aggr_function' to 'aggr_func' for consistency in data processing.
 2025-07-01 15:48:18 -03:00
vitor-aignosi
0e915f5491 SIENTIAPDE-1110 
Update Redis activity to include specific prefixes for keys in last data timestamp and held data, enhancing clarity in data management.
 2025-07-01 15:45:03 -03:00
vitor-aignosi
c9b9974f9d SIENTIAPDE-1110 
Refactor Redis activity to remove unused 'ttl' parameter in set method, streamlining data handling.
 2025-07-01 15:41:49 -03:00
vitor-aignosi
f183bac90d SIENTIAPDE-1110 
Enhance MongoDB data return format by converting to DataFrame dictionary, improving data structure for downstream processing.
 2025-07-01 15:37:40 -03:00
vitor-aignosi
f441873969 SIENTIAPDE-1110 
Refactor MongoDB data retrieval to convert cursor to list, improving data handling and compatibility.
 2025-07-01 15:33:51 -03:00
vitor-aignosi
d7d44556c8 SIENTIAPDE-1110 
Update MongoDB activity to handle None last_data_timestamp by using an empty filter, improving data retrieval logic.
 2025-07-01 15:31:27 -03:00
vitor-aignosi
1908741feb SIENTIAPDE-1110 
Add debug logging for MongoDB data loading to enhance traceability of input data.
 2025-07-01 15:26:57 -03:00
vitor-aignosi
9b524ea428 SIENTIAPDE-1110 
Refactor MongoDB activity to import datetime directly and format 'inserted_at' timestamps to ISO 8601 format for improved data consistency.
 2025-07-01 15:20:50 -03:00
vitor-aignosi
51f4eb1435 SIENTIAPDE-1110 
Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
 2025-07-01 15:16:55 -03:00
vitor-aignosi
97d66ed6cd SIENTIAPDE-1110 
Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
 2025-07-01 15:15:18 -03:00
vitor-aignosi
53337de74a SIENTIAPDE-1110 
SIENTIAPDE-1110 Add MongoDB integration and enhance Redis activity with timestamp management functions.
 2025-07-01 15:10:32 -03:00
vitor-aignosi
e7b05ebf3a SIENTIAPDE-1110 
Add logging for consumer creation in Kafka activity to enhance traceability of topic management.
 2025-07-01 11:05:03 -03:00
vitor-aignosi
946d953598 SIENTIAPDE-1110 
Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
 2025-07-01 10:57:12 -03:00
vitor-aignosi
704099c387 SIENTIAPDE-1110 
Update Kafka activity to use AIOKafkaConsumer for asynchronous message handling, enhancing consumer management and adding support for multiple topics.
 2025-07-01 10:44:37 -03:00
vitor-aignosi
66b568838e SIENTIAPDE-1110 
Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
 2025-07-01 08:21:00 -03:00
vitor-aignosi
a53804331b SIENTIAPDE-1110 
Enhance Kafka activity initialization by adding bootstrap_servers and group_id attributes for improved configuration management.
 2025-06-30 14:36:15 -03:00
vitor-aignosi
d33a5cbb72 SIENTIAPDE-1110 
Refactor Kafka activity to use local kafka_connector variable for improved readability and ensure proper closure of the Kafka consumer after polling.
 2025-06-30 14:12:02 -03:00
vitor-aignosi
6144bcf8ff SIENTIAPDE-1110 
Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
 2025-06-30 14:11:30 -03:00
vitor-aignosi
9afb74be21 SIENTIAPDE-1110 
Add metadata to input_data in Scouter and update CoreScouter to use it
 2025-06-27 10:07:43 -03:00
vitor-aignosi
74a2d9aa3e SIENTIAPDE-1110 
Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
 2025-06-27 09:34:58 -03:00
vitor-aignosi
99d80020d0 Merge pull request #6 from Aignosi/SIENTIAPDE-1110-criar-testes-e-2-e 
Sientiapde 1110 criar testes e 2 e
 2025-06-26 16:15:25 -03:00
vitor-aignosi
f6831ff0aa SIENTIAPDE-1110 
Update image tag in values.yaml from 0.2.1 to 0.2.2
 2025-06-26 15:46:07 -03:00
vitor-aignosi
4a90b77786 SIENTIAPDE-1110 
Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
 2025-06-26 15:15:31 -03:00
vitor-aignosi
9355af1709 SIENTIAPDE-1110 
SIENTIAPDE-1110 Update image tag in values.yaml from 0.1.0-uat to 0.4.0-uat
 2025-06-25 10:07:54 -03:00
vitor-aignosi
92ddaaf17a SIENTIAPDE-1110 
Update image tag in values.yaml from 0.0.2 to 0.2.1
 2025-06-25 09:54:17 -03:00
Bruno Domingues
d31812c7fc SIENTIAPDE-1111: refactor sonarqube pipeline 2. 2025-06-23 22:04:27 -03:00
Bruno Domingues
cff067c517 SIENTIAPDE-1111: refactor sonarqube pipeline. 2025-06-23 21:59:08 -03:00
Matheus Demoner
b6f3e4f813 Merge pull request #5 from Aignosi/SIENTIAPDE-1100-rever-erros-nao-lancados-lancando-os-se-necessario 
SIENTIAPDE-1100
 2025-06-12 10:42:40 -03:00
vitor-aignosi
d27dde99a9 SIENTIAPDE-1100 
Fix error handling in aggregate_data method by raising the caught exception
 2025-06-12 09:27:37 -03:00
Matheus Demoner
7b2a5260b3 Merge pull request #4 from Aignosi/SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious 
SIENTIAPDE-1082
 2025-06-09 16:44:57 -03:00
vitor-aignosi
577513576d Merge branch 'main' into SIENTIAPDE-1082-ajustar-o-worker-e-activities-para-ter-um-destrutor-explicito-como-o-laborious 2025-06-09 15:14:00 -03:00
vitor-aignosi
f541e05e29 Update sientia-mlops-library version to 0.38.1 in requirements.txt 2025-06-09 15:12:43 -03:00
vitor-aignosi
8cf3273714 SIENTIAPDE-1082 
Implement shutdown methods for Activities and Kafka; enhance error handling in worker main function
 2025-06-09 15:09:53 -03:00
Bruno Domingues
ab0c21f5fb Merge pull request #3 from Aignosi/SIENTIAPDE-1094-atualizar-versao-da-lib 
Sientiapde 1094 atualizar versao da lib
 2025-06-09 13:33:51 -03:00
vitor-aignosi
b8beb0e151 SIENTIAPDE-1094 
Refactor Redis activity tests to improve initialization and assertion logic
 2025-06-09 11:06:53 -03:00
vitor-aignosi
3759f10aec SIENTIAPDE-1094 
Refactor test imports and remove unused test files
 2025-06-09 10:37:06 -03:00
vitor-aignosi
7dbb9a29ea SIENTIAPDE-1094 
Update environment configuration and refactor activity imports

- Changed Kafka, Redis, and Temporal host configurations to use localhost.
- Updated the version reference for the sientia-dataops-library in requirements.txt.
- Refactored import paths for activities to align with new module structure.
- Removed unused base.py and postgres.py files.
- Updated logger and policies imports to reflect new module locations.
- Adjusted values.yaml for branch and log level settings.
 2025-06-09 09:43:06 -03:00