2efad30304
SIENTIAPDE-1174 SIENTIAPDE-1174 Update dependencies, modify image tag, and enhance metrics tracking in various activities for improved observability and performance.
vitor-aignosi2025-07-31 12:52:07 -03:00
37d196be14
SIENTIAPDE-1172 Update dependencies in requirements.txt and change image tag and GITHUB_BRANCH in values.yaml for versioning and pipeline adjustments.
vitor-aignosi2025-07-29 13:21:01 -03:00
65efef796f
Merge pull request #12 from Aignosi/SIENTIAPDE-1151-criar-testes-de-stress
Bruno Domingues
2025-07-22 12:22:15 -03:00
3b02a819ad
Merge branch 'main' into SIENTIAPDE-1151-criar-testes-de-stress
vitor-aignosi2025-07-22 11:17:04 -03:00
64a5bb0bbb
Merge pull request #11 from Aignosi/SIENTIAPDE-1163-alterar-dinamica-de-notificacoes-para-usar-o-mongodb-ao-inves-do-kafka
Bruno Domingues
2025-07-22 10:14:10 -03:00
f6913da932
SIENTIAPDE-1151 Enhance test cases in Gates and Redis activities by adding 'name' field to data and updating Redis TTL to 5 hours for better data retention.
vitor-aignosi2025-07-22 08:16:21 -03:00
b68b3b403b
SIENTIAPDE-1151 SIENTIAPDE-1151 Filter data in Gates activity based on model tags to enhance data accuracy.
vitor-aignosi2025-07-21 12:03:03 -03:00
aacd52b5c7
SIENTIAPDE-1151 Add debug logging for collected data in MongoDB activity
vitor-aignosi2025-07-21 11:12:39 -03:00
a9b421b134
SIENTIAPDE-1151 Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1151-criar-testes-de-stress for stress testing implementation.
vitor-aignosi2025-07-18 16:21:18 -03:00
b230ad614e
SIENTIAPDE-1151 Update Redis set method to use a TTL of 5 hours for improved data retention.
vitor-aignosi2025-07-18 16:19:06 -03:00
e6d2bac902
SIENTIAPDE-1163 Remove service configuration from values.yaml to streamline deployment settings.
vitor-aignosi2025-07-17 14:59:14 -03:00
9580b6f8be
SIENTIAPDE-1163 SIENTIAPDE-1163 Update test_redis.py to mock existing data in redis_activity.get for improved error handling in group and hold data tests.
vitor-aignosi2025-07-17 14:30:34 -03:00
839be18644
SIENTIAPDE-1163 Update .gitignore to include .coverage and git_log for better file management
vitor-aignosi2025-07-17 12:33:27 -03:00
b0d0478b20
SIENTIAPDE-1163 Add PR shortcut command to README for easier log generation
vitor-aignosi2025-07-17 12:32:38 -03:00
939acb5354
SIENTIAPDE-1163 Update image tag in values.yaml from 0.2.3 to 0.2.4 for version increment.
vitor-aignosi2025-07-17 12:31:08 -03:00
9329ae206c
SIENTIAPDE-1163 Update sientia-dataops-library dependency version from 1.3.0 to 1.3.3 in requirements.txt and change GITHUB_BRANCH in values.yaml to SIENTIAPDE-1163-alterar-dinamica-de-notificacoes-para-usar-o-mongodb-ao-inves-do-kafka for improved notification handling.
vitor-aignosi2025-07-17 11:00:52 -03:00
03f5518cd1
SIENTIAPDE-1163 Update sientia-dataops-library dependency version from 1.2.1 to 1.3.0 in requirements.txt
vitor-aignosi2025-07-16 09:23:02 -03:00
95defccbf6
SIENTIAPDE-1163 Refactor notification handler imports across multiple files to use CoreNotificationHandler instead of NotificationHandler for consistency and clarity.
vitor-aignosi2025-07-16 08:59:21 -03:00
b014375c3f
SIENTIAPDE-1154 Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1154-otimizar-conexao-com-banco-de-dados-e-paralelismo for enhanced database connection optimization and parallelism.
vitor-aignosi2025-07-15 12:11:20 -03:00
efeef628cb
SIENTIAPDE-1148 Update sonar-project.properties to exclude worker.py from coverage, enhance error handling in worker.py, and add new tests for error scenarios in test_redis.py and test_mongo.py.
vitor-aignosi2025-07-11 12:32:23 -03:00
72c87dfb51
SIENTIAPDE-1148 Update values.yaml to change GITHUB_BRANCH to SIENTIAPDE-1148-separar-scouter-laborious-e-orchestrator-por-namespaces and set TEMPORAL_NAMESPACE to 'scouter' for improved environment configuration.
vitor-aignosi2025-07-11 11:41:59 -03:00
de9b30ada1
Merge pull request #9 from Aignosi/SIENTIAPDE-1141-criar-testes-de-carga
vitor-aignosi2025-07-11 09:24:51 -03:00
91bb8023ca
Merge branch 'main' into SIENTIAPDE-1141-criar-testes-de-carga
vitor-aignosi2025-07-11 08:39:28 -03:00
09d0569998
SIENTIAPDE-1141 Enhance worker configuration in worker.py by adding parameters for max concurrent tasks and cached workflows to improve performance and scalability.
vitor-aignosi2025-07-10 16:06:31 -03:00
b64e1ed833
SIENTIAPDE-1141 Update values.yaml to increase replicaCount from 1 to 5 and change GITHUB_BRANCH value to SIENTIAPDE-1141-criar-testes-de-carga for enhanced testing and scalability.
vitor-aignosi2025-07-10 15:35:33 -03:00
6aa6071fce
SIENTIAPDE-1110 Refactor test cases in test_gates.py to update filter configurations, extracting policy into a dictionary format for improved clarity and consistency in data quality gate tests.
vitor-aignosi2025-07-08 16:37:41 -03:00
a8fe8caf1e
SIENTIAPDE-1110 Update sientia-dataops-library dependency version from 1.2.0 to 1.2.1 in requirements.txt for improved functionality.
vitor-aignosi2025-07-07 16:20:35 -03:00
1404eefdae
SIENTIAPDE-1110 Update Gates activity to extract policy from filter configuration, improving clarity in filter handling.
vitor-aignosi2025-07-07 14:18:40 -03:00
f203ff829d
SIENTIAPDE-1110 SIENTIAPDE-1110 Add store_data_package method to Redis activity for debug data storage functionality.
vitor-aignosi2025-07-07 10:57:52 -03:00
b95cc8ddd5
SIENTIAPDE-1110 Update CoreScouter workflow to include store_data_package method in the main execution flow, enhancing data handling capabilities.
vitor-aignosi2025-07-07 10:56:30 -03:00
38344bbbbf
Merge branch 'main' into SIENTIAPDE-1110-criar-testes-e-2-e
vitor-aignosi2025-07-07 10:03:52 -03:00
0b77a7bc7e
SIENTIAPDE-1110 Implement store_data_package method in Redis activity for debug data storage. Update CoreScouter workflow to conditionally call store_data_package based on debug flag. Enhance tests for store_data_package functionality and error handling.
vitor-aignosi2025-07-07 10:03:08 -03:00
08240efc8b
SIENTIAPDE-1110 Enhance Gates and Redis activities by adding metadata parameter to apply_aggregation and notification methods. Refactor notification handling to use send_notification for improved consistency. Update tests to reflect changes in notification method calls and ensure proper functionality with new metadata integration.
vitor-aignosi2025-07-04 11:01:20 -03:00
d08d1b1337
SIENTIAPDE-1110 Refactor Activities class to remove Kafka and Druid dependencies, simplifying initialization. Update values.yaml to set replica count to 1 for reduced resource usage. Adjust Redis activity to set TTL to None for better data retention. Remove unused Kafka and Druid activity files and their associated tests, streamlining the codebase.
vitor-aignosi2025-07-03 16:25:55 -03:00
1dcba6b48e
SIENTIAPDE-1110 Refactor Scouter workflow to update data loading method, changing from load_latest_druid_data to load_latest_data. Adjusted parameters for consistency in collection naming and improved clarity in data retrieval process.
vitor-aignosi2025-07-03 14:04:10 -03:00
3ee7e3687a
SIENTIAPDE-1110 Update test.ipynb to generate and save multiple pipeline specifications and input samples in JSON format. Modify values.yaml to increase replica count from 1 to 3 and adjust PostgreSQL max connections from 20 to 40 for improved scalability.
vitor-aignosi2025-07-03 14:00:09 -03:00
208491498f
SIENTIAPDE-1110 Update Druid configuration in values.yaml to change port from 8081 to 8888. Refactor pydruid.py to use druid_engine for SQLAlchemy connections, enhancing clarity and consistency in data loading operations.
vitor-aignosi2025-07-02 10:35:53 -03:00
6caec63977
SIENTIAPDE-1110 Refactor last data timestamp handling in Druid activity to use string format for default value, improving consistency in date representation.
vitor-aignosi2025-07-02 10:24:35 -03:00
3bbc6a6a79
SIENTIAPDE-1110 Update Druid activity to handle None values for last data timestamp, ensuring default timestamp is set to epoch start. Enhance data loading debug logging for improved traceability.
vitor-aignosi2025-07-02 10:21:55 -03:00
430fe04202
SIENTIAPDE-1110 Enhance Druid activity by adding configuration for Druid host and port in values.yaml. Refactor data loading logic to utilize SQLAlchemy for querying Druid, improving query efficiency and readability. Update timestamp handling to streamline data retrieval process.
vitor-aignosi2025-07-02 10:19:04 -03:00
879c1fdc49
SIENTIAPDE-1110 Add logging for data loading in Druid activity, enhancing traceability of data retrieval intervals.
vitor-aignosi2025-07-02 09:27:02 -03:00
c3c8e1b766
SIENTIAPDE-1110 Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
vitor-aignosi2025-07-02 09:25:46 -03:00
332612ac84
SIENTIAPDE-1110 Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
vitor-aignosi2025-07-02 09:02:59 -03:00
91f2d8610b
SIENTIAPDE-1110 Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
vitor-aignosi2025-07-01 16:09:46 -03:00
defac7b1f0
SIENTIAPDE-1110 Update Redis activity to include a specific prefix for the last data timestamp key, enhancing clarity in key management.
vitor-aignosi2025-07-01 16:06:52 -03:00
82ce1852e3
SIENTIAPDE-1110 Add debug logging for data filter in MongoDB activity to enhance traceability of filtering criteria.
vitor-aignosi2025-07-01 16:03:40 -03:00
b718801290
SIENTIAPDE-1110 Update get_last_data_timestamp method in Redis activity to return a string or None, improving type clarity for timestamp retrieval.
vitor-aignosi2025-07-01 15:53:47 -03:00
337a54217c
SIENTIAPDE-1110 Fix aggregation function key in Gates activity from 'aggr_function' to 'aggr_func' for consistency in data processing.
vitor-aignosi2025-07-01 15:48:18 -03:00
0e915f5491
SIENTIAPDE-1110 Update Redis activity to include specific prefixes for keys in last data timestamp and held data, enhancing clarity in data management.
vitor-aignosi2025-07-01 15:45:03 -03:00
c9b9974f9d
SIENTIAPDE-1110 Refactor Redis activity to remove unused 'ttl' parameter in set method, streamlining data handling.
vitor-aignosi2025-07-01 15:41:49 -03:00
f183bac90d
SIENTIAPDE-1110 Enhance MongoDB data return format by converting to DataFrame dictionary, improving data structure for downstream processing.
vitor-aignosi2025-07-01 15:37:40 -03:00
f441873969
SIENTIAPDE-1110 Refactor MongoDB data retrieval to convert cursor to list, improving data handling and compatibility.
vitor-aignosi2025-07-01 15:33:51 -03:00
d7d44556c8
SIENTIAPDE-1110 Update MongoDB activity to handle None last_data_timestamp by using an empty filter, improving data retrieval logic.
vitor-aignosi2025-07-01 15:31:27 -03:00
1908741feb
SIENTIAPDE-1110 Add debug logging for MongoDB data loading to enhance traceability of input data.
vitor-aignosi2025-07-01 15:26:57 -03:00
9b524ea428
SIENTIAPDE-1110 Refactor MongoDB activity to import datetime directly and format 'inserted_at' timestamps to ISO 8601 format for improved data consistency.
vitor-aignosi2025-07-01 15:20:50 -03:00
51f4eb1435
SIENTIAPDE-1110 Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
vitor-aignosi2025-07-01 15:16:55 -03:00
97d66ed6cd
SIENTIAPDE-1110 Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
vitor-aignosi2025-07-01 15:15:18 -03:00
53337de74a
SIENTIAPDE-1110 SIENTIAPDE-1110 Add MongoDB integration and enhance Redis activity with timestamp management functions.
vitor-aignosi2025-07-01 15:10:32 -03:00
e7b05ebf3a
SIENTIAPDE-1110 Add logging for consumer creation in Kafka activity to enhance traceability of topic management.
vitor-aignosi2025-07-01 11:05:03 -03:00
946d953598
SIENTIAPDE-1110 Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
vitor-aignosi2025-07-01 10:57:12 -03:00
704099c387
SIENTIAPDE-1110 Update Kafka activity to use AIOKafkaConsumer for asynchronous message handling, enhancing consumer management and adding support for multiple topics.
vitor-aignosi2025-07-01 10:44:37 -03:00
66b568838e
SIENTIAPDE-1110 Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
vitor-aignosi2025-07-01 08:21:00 -03:00
a53804331b
SIENTIAPDE-1110 Enhance Kafka activity initialization by adding bootstrap_servers and group_id attributes for improved configuration management.
vitor-aignosi2025-06-30 14:36:15 -03:00
d33a5cbb72
SIENTIAPDE-1110 Refactor Kafka activity to use local kafka_connector variable for improved readability and ensure proper closure of the Kafka consumer after polling.
vitor-aignosi2025-06-30 14:12:02 -03:00
6144bcf8ff
SIENTIAPDE-1110 Update Kafka polling time in values.yaml and refactor Kafka subscription in kafka.py to use KafkaConsumer for improved message handling.
vitor-aignosi2025-06-30 14:11:30 -03:00
9afb74be21
SIENTIAPDE-1110 Add metadata to input_data in Scouter and update CoreScouter to use it
vitor-aignosi2025-06-27 10:07:43 -03:00
74a2d9aa3e
SIENTIAPDE-1110 Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
vitor-aignosi2025-06-27 09:34:58 -03:00
99d80020d0
Merge pull request #6 from Aignosi/SIENTIAPDE-1110-criar-testes-e-2-e
vitor-aignosi2025-06-26 16:15:25 -03:00
f6831ff0aa
SIENTIAPDE-1110 Update image tag in values.yaml from 0.2.1 to 0.2.2
vitor-aignosi2025-06-26 15:46:07 -03:00
4a90b77786
SIENTIAPDE-1110 Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
vitor-aignosi2025-06-26 15:15:31 -03:00
9355af1709
SIENTIAPDE-1110 SIENTIAPDE-1110 Update image tag in values.yaml from 0.1.0-uat to 0.4.0-uat
vitor-aignosi2025-06-25 10:07:54 -03:00
92ddaaf17a
SIENTIAPDE-1110 Update image tag in values.yaml from 0.0.2 to 0.2.1
vitor-aignosi2025-06-25 09:54:17 -03:00
d31812c7fc
SIENTIAPDE-1111: refactor sonarqube pipeline 2.
Bruno Domingues2025-06-23 22:04:27 -03:00
vitor-aignosi
Bruno Domingues
Bruno Domingues