95defccbf6
SIENTIAPDE-1163 Refactor notification handler imports across multiple files to use CoreNotificationHandler instead of NotificationHandler for consistency and clarity.
vitor-aignosi
2025-07-16 08:59:21 -03:00
b014375c3f
SIENTIAPDE-1154 Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1154-otimizar-conexao-com-banco-de-dados-e-paralelismo for enhanced database connection optimization and parallelism.
vitor-aignosi
2025-07-15 12:11:20 -03:00
efeef628cb
SIENTIAPDE-1148 Update sonar-project.properties to exclude worker.py from coverage, enhance error handling in worker.py, and add new tests for error scenarios in test_redis.py and test_mongo.py.
vitor-aignosi
2025-07-11 12:32:23 -03:00
72c87dfb51
SIENTIAPDE-1148 Update values.yaml to change GITHUB_BRANCH to SIENTIAPDE-1148-separar-scouter-laborious-e-orchestrator-por-namespaces and set TEMPORAL_NAMESPACE to 'scouter' for improved environment configuration.
vitor-aignosi
2025-07-11 11:41:59 -03:00
91bb8023ca
Merge branch 'main' into SIENTIAPDE-1141-criar-testes-de-carga
vitor-aignosi
2025-07-11 08:39:28 -03:00
09d0569998
SIENTIAPDE-1141 Enhance worker configuration in worker.py by adding parameters for max concurrent tasks and cached workflows to improve performance and scalability.
vitor-aignosi
2025-07-10 16:06:31 -03:00
b64e1ed833
SIENTIAPDE-1141 Update values.yaml to increase replicaCount from 1 to 5 and change GITHUB_BRANCH value to SIENTIAPDE-1141-criar-testes-de-carga for enhanced testing and scalability.
vitor-aignosi
2025-07-10 15:35:33 -03:00
6aa6071fce
SIENTIAPDE-1110 Refactor test cases in test_gates.py to update filter configurations, extracting policy into a dictionary format for improved clarity and consistency in data quality gate tests.
vitor-aignosi
2025-07-08 16:37:41 -03:00
a8fe8caf1e
SIENTIAPDE-1110 Update sientia-dataops-library dependency version from 1.2.0 to 1.2.1 in requirements.txt for improved functionality.
vitor-aignosi
2025-07-07 16:20:35 -03:00
1404eefdae
SIENTIAPDE-1110 Update Gates activity to extract policy from filter configuration, improving clarity in filter handling.
vitor-aignosi
2025-07-07 14:18:40 -03:00
f203ff829d
SIENTIAPDE-1110 SIENTIAPDE-1110 Add store_data_package method to Redis activity for debug data storage functionality.
vitor-aignosi
2025-07-07 10:57:52 -03:00
b95cc8ddd5
SIENTIAPDE-1110 Update CoreScouter workflow to include store_data_package method in the main execution flow, enhancing data handling capabilities.
vitor-aignosi
2025-07-07 10:56:30 -03:00
38344bbbbf
Merge branch 'main' into SIENTIAPDE-1110-criar-testes-e-2-e
vitor-aignosi
2025-07-07 10:03:52 -03:00
0b77a7bc7e
SIENTIAPDE-1110 Implement store_data_package method in Redis activity for debug data storage. Update CoreScouter workflow to conditionally call store_data_package based on debug flag. Enhance tests for store_data_package functionality and error handling.
vitor-aignosi
2025-07-07 10:03:08 -03:00
08240efc8b
SIENTIAPDE-1110 Enhance Gates and Redis activities by adding metadata parameter to apply_aggregation and notification methods. Refactor notification handling to use send_notification for improved consistency. Update tests to reflect changes in notification method calls and ensure proper functionality with new metadata integration.
vitor-aignosi
2025-07-04 11:01:20 -03:00
d08d1b1337
SIENTIAPDE-1110 Refactor Activities class to remove Kafka and Druid dependencies, simplifying initialization. Update values.yaml to set replica count to 1 for reduced resource usage. Adjust Redis activity to set TTL to None for better data retention. Remove unused Kafka and Druid activity files and their associated tests, streamlining the codebase.
vitor-aignosi
2025-07-03 16:25:55 -03:00
1dcba6b48e
SIENTIAPDE-1110 Refactor Scouter workflow to update data loading method, changing from load_latest_druid_data to load_latest_data. Adjusted parameters for consistency in collection naming and improved clarity in data retrieval process.
vitor-aignosi
2025-07-03 14:04:10 -03:00
3ee7e3687a
SIENTIAPDE-1110 Update test.ipynb to generate and save multiple pipeline specifications and input samples in JSON format. Modify values.yaml to increase replica count from 1 to 3 and adjust PostgreSQL max connections from 20 to 40 for improved scalability.
vitor-aignosi
2025-07-03 14:00:09 -03:00
208491498f
SIENTIAPDE-1110 Update Druid configuration in values.yaml to change port from 8081 to 8888. Refactor pydruid.py to use druid_engine for SQLAlchemy connections, enhancing clarity and consistency in data loading operations.
vitor-aignosi
2025-07-02 10:35:53 -03:00
6caec63977
SIENTIAPDE-1110 Refactor last data timestamp handling in Druid activity to use string format for default value, improving consistency in date representation.
vitor-aignosi
2025-07-02 10:24:35 -03:00
3bbc6a6a79
SIENTIAPDE-1110 Update Druid activity to handle None values for last data timestamp, ensuring default timestamp is set to epoch start. Enhance data loading debug logging for improved traceability.
vitor-aignosi
2025-07-02 10:21:55 -03:00
430fe04202
SIENTIAPDE-1110 Enhance Druid activity by adding configuration for Druid host and port in values.yaml. Refactor data loading logic to utilize SQLAlchemy for querying Druid, improving query efficiency and readability. Update timestamp handling to streamline data retrieval process.
vitor-aignosi
2025-07-02 10:19:04 -03:00
879c1fdc49
SIENTIAPDE-1110 Add logging for data loading in Druid activity, enhancing traceability of data retrieval intervals.
vitor-aignosi
2025-07-02 09:27:02 -03:00
c3c8e1b766
SIENTIAPDE-1110 Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
vitor-aignosi
2025-07-02 09:25:46 -03:00
332612ac84
SIENTIAPDE-1110 Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
vitor-aignosi
2025-07-02 09:02:59 -03:00
91f2d8610b
SIENTIAPDE-1110 Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
vitor-aignosi
2025-07-01 16:09:46 -03:00
defac7b1f0
SIENTIAPDE-1110 Update Redis activity to include a specific prefix for the last data timestamp key, enhancing clarity in key management.
vitor-aignosi
2025-07-01 16:06:52 -03:00
82ce1852e3
SIENTIAPDE-1110 Add debug logging for data filter in MongoDB activity to enhance traceability of filtering criteria.
vitor-aignosi
2025-07-01 16:03:40 -03:00
b718801290
SIENTIAPDE-1110 Update get_last_data_timestamp method in Redis activity to return a string or None, improving type clarity for timestamp retrieval.
vitor-aignosi
2025-07-01 15:53:47 -03:00
337a54217c
SIENTIAPDE-1110 Fix aggregation function key in Gates activity from 'aggr_function' to 'aggr_func' for consistency in data processing.
vitor-aignosi
2025-07-01 15:48:18 -03:00
0e915f5491
SIENTIAPDE-1110 Update Redis activity to include specific prefixes for keys in last data timestamp and held data, enhancing clarity in data management.
vitor-aignosi
2025-07-01 15:45:03 -03:00
c9b9974f9d
SIENTIAPDE-1110 Refactor Redis activity to remove unused 'ttl' parameter in set method, streamlining data handling.
vitor-aignosi
2025-07-01 15:41:49 -03:00
f183bac90d
SIENTIAPDE-1110 Enhance MongoDB data return format by converting to DataFrame dictionary, improving data structure for downstream processing.
vitor-aignosi
2025-07-01 15:37:40 -03:00
f441873969
SIENTIAPDE-1110 Refactor MongoDB data retrieval to convert cursor to list, improving data handling and compatibility.
vitor-aignosi
2025-07-01 15:33:51 -03:00
d7d44556c8
SIENTIAPDE-1110 Update MongoDB activity to handle None last_data_timestamp by using an empty filter, improving data retrieval logic.
vitor-aignosi
2025-07-01 15:31:27 -03:00
1908741feb
SIENTIAPDE-1110 Add debug logging for MongoDB data loading to enhance traceability of input data.
vitor-aignosi
2025-07-01 15:26:57 -03:00
9b524ea428
SIENTIAPDE-1110 Refactor MongoDB activity to import datetime directly and format 'inserted_at' timestamps to ISO 8601 format for improved data consistency.
vitor-aignosi
2025-07-01 15:20:50 -03:00
51f4eb1435
SIENTIAPDE-1110 Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
vitor-aignosi
2025-07-01 15:16:55 -03:00
97d66ed6cd
SIENTIAPDE-1110 Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
vitor-aignosi
2025-07-01 15:15:18 -03:00
53337de74a
SIENTIAPDE-1110 SIENTIAPDE-1110 Add MongoDB integration and enhance Redis activity with timestamp management functions.
vitor-aignosi
2025-07-01 15:10:32 -03:00
e7b05ebf3a
SIENTIAPDE-1110 Add logging for consumer creation in Kafka activity to enhance traceability of topic management.
vitor-aignosi
2025-07-01 11:05:03 -03:00
946d953598
SIENTIAPDE-1110 Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
vitor-aignosi
2025-07-01 10:57:12 -03:00
704099c387
SIENTIAPDE-1110 Update Kafka activity to use AIOKafkaConsumer for asynchronous message handling, enhancing consumer management and adding support for multiple topics.
vitor-aignosi
2025-07-01 10:44:37 -03:00
66b568838e
SIENTIAPDE-1110 Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
vitor-aignosi
2025-07-01 08:21:00 -03:00
a53804331b
SIENTIAPDE-1110 Enhance Kafka activity initialization by adding bootstrap_servers and group_id attributes for improved configuration management.
vitor-aignosi
2025-06-30 14:36:15 -03:00
d33a5cbb72
SIENTIAPDE-1110 Refactor Kafka activity to use local kafka_connector variable for improved readability and ensure proper closure of the Kafka consumer after polling.
vitor-aignosi
2025-06-30 14:12:02 -03:00
6144bcf8ff
SIENTIAPDE-1110 Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
vitor-aignosi
2025-06-30 14:11:30 -03:00
9afb74be21
SIENTIAPDE-1110 Add metadata to input_data in Scouter and update CoreScouter to use it
vitor-aignosi
2025-06-27 10:07:43 -03:00
74a2d9aa3e
SIENTIAPDE-1110 Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
vitor-aignosi
2025-06-27 09:34:58 -03:00
f6831ff0aa
SIENTIAPDE-1110 Update image tag in values.yaml from 0.2.1 to 0.2.2
vitor-aignosi
2025-06-26 15:46:07 -03:00
4a90b77786
SIENTIAPDE-1110 Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
vitor-aignosi
2025-06-26 15:15:31 -03:00
9355af1709
SIENTIAPDE-1110 SIENTIAPDE-1110 Update image tag in values.yaml from 0.1.0-uat to 0.4.0-uat
vitor-aignosi
2025-06-25 10:07:54 -03:00
92ddaaf17a
SIENTIAPDE-1110 Update image tag in values.yaml from 0.0.2 to 0.2.1
vitor-aignosi
2025-06-25 09:54:17 -03:00
vitor-aignosi