5dc2971d8c
SIENTIAPDE-1314 SIENTIAPDE-1314: Update workflow name from 'sub_core_scouter' to 'subworkflow.core_scouter' in scouter.py and core_scouter.py, and adjust test references accordingly in test_scouter.py for consistency.
vitor-aignosi
2025-10-27 10:16:57 -03:00
6af45c7476
SIENTIAPDE-1314 SIENTIAPDE-1314: Rename core_scouter workflow to sub_core_scouter and update references in scouter.py and test_scouter.py for consistency.
vitor-aignosi
2025-10-27 10:11:07 -03:00
25838c26a6
SIENTIAPDE-1314 Update GITHUB_BRANCH in values.yaml to fix/SIENTIAPDE-1314 for monitoring layer adjustments
vitor-aignosi
2025-10-24 10:14:26 -03:00
d1521838c5
SIENTIAPDE-1314 SIENTIAPDE-1314: Update type annotations in encrypt.py for improved clarity and type safety. Added type hint for file_tree variable and included type ignore for yaml import.
vitor-aignosi
2025-10-23 16:15:29 -03:00
05fa808bfd
SIENTIAPDE-1314 SIENTIAPDE-1314: Remove redundant blank line in Redis class to enhance code clarity.
vitor-aignosi
2025-10-23 16:10:36 -03:00
fb1440b0d7
SIENTIAPDE-1314 Remove unnecessary blank line in Redis class definition to improve code readability.
vitor-aignosi
2025-10-23 16:08:44 -03:00
f0ca06b4aa
SIENTIAPDE-1314 Implement tag value registration in Gates activity and update related tests. Enhanced the write_metrics method to process and register tag values, improving metrics tracking. Adjusted tests to validate new functionality.
vitor-aignosi
2025-10-23 16:02:38 -03:00
c9136ca537
Merge pull request #22 from Aignosi/feature/SIENTIAPDE-1327-controle-de-releases
Bruno Domingues
2025-10-23 10:45:15 -03:00
b063f973cd
SIENTIAPDE-1327 Update requirements and simplify CI workflow by integrating quality gate from template. Bump sientia-dataops-library version to 1.4.7 and streamline quality-gate.yml to use a shared workflow configuration.
vitor-aignosi
2025-10-23 09:04:28 -03:00
507ee8cc18
SIENTIAPDE-1312 Update image tag to 0.4.9 and change GITHUB_BRANCH to SIENTIAPDE-1312-melhorias-e-correcoes-nas-pipelines-de-dados in values.yaml
vitor-aignosi
2025-10-21 14:40:07 -03:00
9cc55ea5e7
SIENTIAPDE-1312 Update Redis and CoreScouter to incorporate 'fill_missing_tags' parameter in data processing. Adjusted related tests to ensure proper handling of missing tags, enhancing overall functionality and consistency across workflows.
vitor-aignosi
2025-10-21 09:14:41 -03:00
bd6474a975
SIENTIAPDE-1312 SIENTIAPDE-1318: Add 'fill_missing_tags' parameter to CoreScouter for enhanced tag handling in data processing. This update allows for the specification of missing tags during workflow execution.
vitor-aignosi
2025-10-21 09:01:08 -03:00
4bd29ae0d7
SIENTIAPDE-1318 SIENTIAPDE-1318: Add support for filling missing tags in Redis data processing. Enhanced the Redis class to include a new parameter for handling missing tags, ensuring that absent tags are filled with None in the data package.
vitor-aignosi
2025-10-20 16:24:36 -03:00
ecfdb4873d
SIENTIAPDE-1316 fix on requirements
vitor-aignosi
2025-10-16 15:02:25 -03:00
97eb5bc904
SIENTIAPDE-1316 Update .gitignore and refactor metrics.py, activities.py, and gates.py for improved clarity and consistency. Added coverage.xml and cache directories to .gitignore. Standardized string formatting and parameter handling in metrics and activities classes, enhancing code readability. Removed the deprecated faker.py file and adjusted related tests accordingly.
vitor-aignosi
2025-10-16 13:31:12 -03:00
8bdbf049b8
SIENTIAPDE-1316 SIENTIAPDE-1084: Refactor apply_aggregation method in Gates class for improved performance and clarity. Changed parameter name from 'group' to 'values', optimized aggregation logic, and enhanced error handling for invalid functions. Streamlined data processing by reducing unnecessary operations and added batch debug logging.
vitor-aignosi
2025-10-16 13:06:44 -03:00
205284d9f4
Merge pull request #18 from Aignosi/SIENTIAPDE-1084-ajustar-documentacao
Bruno Domingues
2025-09-03 13:33:41 +00:00
199b1deacb
SIENTIAPDE-1084 Update test_activities.py to change assertions for mock_postgres_close and mock_mongodb_close from assert_called_once to assert_called, allowing for more flexible verification of calls during shutdown tests.
vitor-aignosi
2025-09-01 16:07:42 -03:00
1a16a17267
SIENTIAPDE-1084 Remove quality filters from the CoreScouter Workflow flowchart in README.md to enhance clarity and focus on the main data processing steps.
vitor-aignosi
2025-08-29 16:57:27 -03:00
4c929fe866
SIENTIAPDE-1084 Remove outdated key components section from README.md to streamline the documentation and focus on the CoreScouter Workflow.
vitor-aignosi
2025-08-29 16:55:42 -03:00
da15f2d40d
SIENTIAPDE-1084 Update README.md to modify the Scouter Workflow flowchart, replacing the icon for the core_scouter step with a refresh symbol for better visual representation of data processing continuity.
vitor-aignosi
2025-08-29 16:55:21 -03:00
864bf0af96
SIENTIAPDE-1084 Update README.md to correct the flowchart syntax for the Scouter Workflow Architecture, changing from 'flowchart-elk' to 'flowchart' to ensure proper rendering and maintain clarity in visualizing data processing steps.
vitor-aignosi
2025-08-29 16:43:05 -03:00
abdec12635
SIENTIAPDE-1084 Update README.md to modify the Scouter Workflow Architecture flowchart, changing the layout to 'flowchart-elk' for improved clarity in visualizing data processing steps and interactions.
vitor-aignosi
2025-08-29 16:42:33 -03:00
e573923bdd
SIENTIAPDE-1084 Refactor README.md to update the Scouter Workflow Architecture flowchart, transitioning to a left-to-right layout for improved readability and clarity in depicting data processing steps and service interactions.
vitor-aignosi
2025-08-29 16:41:09 -03:00
791d8b8be2
SIENTIAPDE-1084 Update README.md to revise the Scouter Workflow Architecture flowchart, replacing outdated steps with a new data processing sequence that includes data quality checks, aggregation, and metrics writing, enhancing clarity on system operations and service interactions.
vitor-aignosi
2025-08-29 16:39:31 -03:00
6c22b00d5f
SIENTIAPDE-1084 Update README.md to enhance the architecture diagrams for both Scouter and FakeData workflows, improving clarity and structure by adopting a consistent flowchart format and detailing service interactions.
vitor-aignosi
2025-08-29 16:37:34 -03:00
79dc20e993
SIENTIAPDE-1084 Update README.md to modify the Scouter Workflow Architecture flowchart, enhancing clarity by restructuring the diagram layout and correcting the connections to Redis and MongoDB components.
vitor-aignosi
2025-08-29 16:31:02 -03:00
f5c4d38a7a
SIENTIAPDE-1084 Refactor README.md to streamline the Scouter Workflow Architecture section by consolidating the flowchart into a single line format for improved readability and clarity.
vitor-aignosi
2025-08-29 16:28:49 -03:00
cbad5df8d2
SIENTIAPDE-1084 Update README.md to include a new section on Scouter Workflow Architecture, detailing the incremental data processing steps and key components, while removing outdated system architecture diagrams for improved clarity and relevance.
vitor-aignosi
2025-08-29 16:27:21 -03:00
fe6133f31a
SIENTIAPDE-1084 Enhance README.md by adding detailed system architecture and data processing flow diagrams using Mermaid syntax, improving clarity on the Scouter system's components and interactions.
vitor-aignosi
2025-08-29 16:20:07 -03:00
beb06d257f
SIENTIAPDE-1084 Update README.md to change the section title from "Temporal input parameters" to "MongoDB pipeline configuration" for improved accuracy in documentation.
vitor-aignosi
2025-08-29 16:04:07 -03:00
cf1ce1a38d
SIENTIAPDE-1084 Enhance README.md by adding a section for Scouter Workflow Input Parameters to improve documentation clarity on system configuration.
vitor-aignosi
2025-08-29 16:02:34 -03:00
9dc3a4c55a
SIENTIAPDE-1084 Update README.md to reflect new workflow configuration parameters, including temporal input settings, tag definitions, and filter policies for enhanced clarity on system capabilities.
vitor-aignosi
2025-08-29 15:19:32 -03:00
88576252af
SIENTIAPDE-1084 Update .gitignore to include git_log and .env files for better environment management.
vitor-aignosi
2025-08-29 11:57:05 -03:00
a973da9d60
SIENTIAPDE-1084 Remove deprecated files and configurations, including .env, Dockerfile, docker-compose.yml, and client-schedule.py. Update README.md to reflect new architecture and features, enhancing clarity on system capabilities and workflows. Adjust values.yaml for image tag and replica count, and improve code documentation across various modules for better maintainability.
vitor-aignosi
2025-08-29 11:56:45 -03:00
00d25bdefc
Merge pull request #17 from Aignosi/SIENTIAPDE-1193-conferir-como-a-escrita-de-datetime-ocorre-no-temporal
Bruno Domingues
2025-08-26 13:39:06 +00:00
730c9e9ff9
Merge branch 'main' into SIENTIAPDE-1193-conferir-como-a-escrita-de-datetime-ocorre-no-temporal
vitor-aignosi
2025-08-25 10:46:36 -03:00
719e3917ab
SIENTIAPDE-1193 Refactor Redis key format in tests for last data timestamp to use colons, enhancing readability and consistency with recent changes.
vitor-aignosi
2025-08-25 09:03:03 -03:00
5c01d024a8
SIENTIAPDE-1193 Update image tag in values.yaml from 0.4.3 to 0.4.4 for version increment.
vitor-aignosi
2025-08-25 08:28:12 -03:00
18f7ab6723
SIENTIAPDE-1193 Update replica count in values.yaml from 1 to 3 to scale application deployment.
vitor-aignosi
2025-08-22 15:46:14 -03:00
826b559489
SIENTIAPDE-1193 Refactor CoreScouter workflow test to use updated DATETIME_FORMAT_WITH_TZ, ensuring consistency with recent changes in timestamp handling.
vitor-aignosi
2025-08-22 15:44:13 -03:00
7bbfa4dbac
SIENTIAPDE-1193 Refactor timestamp handling in CoreScouter workflow by updating to DATETIME_FORMAT_WITH_TZ, ensuring consistency with recent changes in datetime management.
vitor-aignosi
2025-08-22 15:33:41 -03:00
54c3dd25e9
SIENTIAPDE-1193 Update replica count in values.yaml to 1 and enhance timestamp handling in MongoDB activity by setting timezone to UTC for inserted_at field.
vitor-aignosi
2025-08-22 15:30:47 -03:00
39ba82b539
SIENTIAPDE-1193 Update image tag to 0.4.3 in values.yaml and refactor Redis key format for last data timestamp to use colons for improved readability.
vitor-aignosi
2025-08-22 15:13:04 -03:00
96a28e91e5
SIENTIAPDE-1193 Enhance timestamp handling in Redis and CoreScouter workflows by updating to DATETIME_FORMAT_MS_WITH_TZ. Modify tests to reflect new timestamp format, ensuring consistency across data structures and improving overall datetime management.
vitor-aignosi
2025-08-22 14:05:41 -03:00
0b337d00f9
SIENTIAPDE-1193 Refactor Redis activity to utilize the new 'now' function for timestamp generation, ensuring consistency with updated datetime handling. This change replaces the direct use of 'datetime.now()' with 'now()' from the temporal constants.
vitor-aignosi
2025-08-22 11:24:26 -03:00
0e0603d2bc
SIENTIAPDE-1193 Update sientia-dataops-library version to 1.4.3 and refactor timestamp handling in MongoDB and Redis activities to use DATETIME_FORMAT constants for improved consistency. Enhance unit tests to reflect these changes.
vitor-aignosi
2025-08-22 11:12:37 -03:00
eabc642feb
SIENTIAPDE-1193 Update dependencies in requirements.txt and change GITHUB_BRANCH in values.yaml to reflect new datetime handling task.
vitor-aignosi
2025-08-22 08:02:13 -03:00
fad10bc0d0
SIENTIAPDE-1193 Update requirements and core scouter workflow: comment out old dataops library dependency, change default namespace in worker.py, and enhance timestamp handling in core_scouter.py with DATETIME_FORMAT.
vitor-aignosi
2025-08-21 12:16:06 -03:00
799aa5af3e
SIENTIAPDE-1199 Refactor logging in worker.py to use custom_info and custom_error methods, enhancing log metadata with pod and workflow details for improved observability.
vitor-aignosi
2025-08-21 11:35:26 -03:00
76d96c2c13
Merge branch 'main' into SIENTIAPDE-1199-revisar-e-testar-observabilidade
vitor-aignosi
2025-08-21 08:36:38 -03:00
2a76f2e3c6
SIENTIAPDE-1199 Enhance logging in Gates, MongoDB, and Redis activities by replacing debug statements with info level logs, improving observability of data processing steps. Update worker configuration to adjust concurrency settings and enable autoscaling for task polling.
vitor-aignosi
2025-08-20 12:39:43 -03:00
3bbc2c4993
SIENTIAPDE-1199 Update dependencies and improve observability by changing logger imports. Bump sientia-dataops-library version to 1.4.1 and update image tag to 0.4.2 in values.yaml.
vitor-aignosi
2025-08-19 16:29:30 -03:00
552f58ca86
SIENTIAPDE-1169 SIENTIAPDE-1169 Update Redis and core scouter tests to include model_tags in data structures, enhancing data organization and consistency across test cases.
vitor-aignosi
2025-08-19 09:29:46 -03:00
2c0f5a67e6
Merge branch 'main' into SIENTIAPDE-1169-pensar-e-projetar-testes-de-breakdown-e-performance
vitor-aignosi
2025-08-18 15:20:23 -03:00
9480495155
SIENTIAPDE-1169 Enable SDK metrics service in values.yaml and update worker.py to initialize SDK metrics server with Prometheus configuration, enhancing observability of the application.
vitor-aignosi
2025-08-18 13:45:20 -03:00
4bbd937b7f
SIENTIAPDE-1169 Enhance Redis activity logging by adding debug statements for tag management and metrics registration, improving traceability of data processing.
vitor-aignosi
2025-08-14 16:33:39 -03:00
9303158a9d
SIENTIAPDE-1169 Update image tag to 0.4.0 in values.yaml, modify GITHUB_BRANCH for SIENTIAPDE-1169, and enhance Redis activity to handle model tags for improved data retention.
vitor-aignosi
2025-08-14 16:15:20 -03:00
77b65bf740
Merge pull request #14 from Aignosi/SIENTIAPDE-1174-mapear-e-implementar-metricas-a-serem-criadas
Bruno Domingues
2025-08-13 16:34:49 -03:00
ba994f9bb4
Merge branch 'main' into SIENTIAPDE-1174-mapear-e-implementar-metricas-a-serem-criadas
vitor-aignosi
2025-08-05 14:04:46 -03:00
c6fca2a8c2
SIENTIAPDE-1174 Update image tag to 0.3.2 in values.yaml for version consistency in deployment.
vitor-aignosi
2025-08-04 15:26:21 -03:00
d4718c7d95
SIENTIAPDE-1174 Update Helm upgrade command in values.yaml to use version 0.4.0, ensuring consistency in deployment instructions.
vitor-aignosi
2025-08-04 10:04:01 -03:00
07c54e9edd
Merge pull request #13 from Aignosi/SIENTIAPDE-1172-criar-pipeline-de-alertas-orquestrador
Bruno Domingues
2025-08-04 10:02:39 -03:00
25f9d89317
SIENTIAPDE-1174 Update image tag to 0.3.1 in values.yaml and modify write_metrics method signature in Gates activity for improved type clarity.
vitor-aignosi
2025-08-01 08:36:08 -03:00
40c57ed23d
SIENTIAPDE-1174 Implement async test for write_metrics method in Gates activity, enhancing test coverage for metric logging functionality.
vitor-aignosi
2025-08-01 07:40:06 -03:00
fd740ca451
SIENTIAPDE-1174 Update Gates activity to use labels in LABORIOUS_DATA_WRITTEN_COUNT metric, improving clarity in metric tracking.
vitor-aignosi
2025-07-31 16:34:55 -03:00
7ea9c1ce65
SIENTIAPDE-1174 Update Gates and Redis activities to use 'workflow_name' instead of 'pipeline_name' in metric tracking, ensuring consistency across the codebase.
vitor-aignosi
2025-07-31 16:31:38 -03:00
b9c004827f
SIENTIAPDE-1174 Update Redis activity to use metadata for model and pipeline names, improving consistency in metric tracking.
vitor-aignosi
2025-07-31 16:27:18 -03:00
a8222359cc
SIENTIAPDE-1174 Refactor Scouter class to remove pod_id reference, enhancing code clarity and consistency in data handling.
vitor-aignosi
2025-07-31 16:25:02 -03:00
205810e882
SIENTIAPDE-1174 Update requirements for sientia-dataops-library to version 1.3.7, add ServiceMonitor configuration for Prometheus in values.yaml, and refactor pod_id usage in Gates and Redis activities for improved consistency in metric tracking.
vitor-aignosi
2025-07-31 15:45:18 -03:00
c69ae5df03
SIENTIAPDE-1174 Remove unused getenv import from scouter.py to enhance code clarity and maintainability.
vitor-aignosi
2025-07-31 14:59:12 -03:00
5d3eb74022
SIENTIAPDE-1174 SIENTIAPDE-1174 Add pod_id initialization in Activities class and update Scouter to reference it for improved consistency in environment variable handling.
vitor-aignosi
2025-07-31 14:57:22 -03:00
ce20d86328
SIENTIAPDE-1174 Remove unused metrics import from core_scouter.py to streamline code and improve clarity.
vitor-aignosi
2025-07-31 14:48:00 -03:00
b4827935a7
SIENTIAPDE-1174 Integrate write_metrics activity into CoreScouter for enhanced metric logging and update worker.py to include write_metrics in workflow tasks.
vitor-aignosi
2025-07-31 14:43:34 -03:00
1c78cfb558
SIENTIAPDE-1174 Add write_metrics activity to Gates for database metric logging and import metrics module
vitor-aignosi
2025-07-31 14:39:58 -03:00
7925b8a2ae
SIENTIAPDE-1174 Update replica count in values.yaml, add application status metrics in metrics.py, and implement Prometheus server in worker.py for enhanced observability.
vitor-aignosi
2025-07-31 13:01:39 -03:00
2efad30304
SIENTIAPDE-1174 SIENTIAPDE-1174 Update dependencies, modify image tag, and enhance metrics tracking in various activities for improved observability and performance.
vitor-aignosi
2025-07-31 12:52:07 -03:00
37d196be14
SIENTIAPDE-1172 Update dependencies in requirements.txt and change image tag and GITHUB_BRANCH in values.yaml for versioning and pipeline adjustments.
vitor-aignosi
2025-07-29 13:21:01 -03:00
65efef796f
Merge pull request #12 from Aignosi/SIENTIAPDE-1151-criar-testes-de-stress
Bruno Domingues
2025-07-22 12:22:15 -03:00
3b02a819ad
Merge branch 'main' into SIENTIAPDE-1151-criar-testes-de-stress
vitor-aignosi
2025-07-22 11:17:04 -03:00
64a5bb0bbb
Merge pull request #11 from Aignosi/SIENTIAPDE-1163-alterar-dinamica-de-notificacoes-para-usar-o-mongodb-ao-inves-do-kafka
Bruno Domingues
2025-07-22 10:14:10 -03:00
f6913da932
SIENTIAPDE-1151 Enhance test cases in Gates and Redis activities by adding 'name' field to data and updating Redis TTL to 5 hours for better data retention.
vitor-aignosi
2025-07-22 08:16:21 -03:00
b68b3b403b
SIENTIAPDE-1151 SIENTIAPDE-1151 Filter data in Gates activity based on model tags to enhance data accuracy.
vitor-aignosi
2025-07-21 12:03:03 -03:00
aacd52b5c7
SIENTIAPDE-1151 Add debug logging for collected data in MongoDB activity
vitor-aignosi
2025-07-21 11:12:39 -03:00
a9b421b134
SIENTIAPDE-1151 Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1151-criar-testes-de-stress for stress testing implementation.
vitor-aignosi
2025-07-18 16:21:18 -03:00
b230ad614e
SIENTIAPDE-1151 Update Redis set method to use a TTL of 5 hours for improved data retention.
vitor-aignosi
2025-07-18 16:19:06 -03:00
e6d2bac902
SIENTIAPDE-1163 Remove service configuration from values.yaml to streamline deployment settings.
vitor-aignosi
2025-07-17 14:59:14 -03:00
9580b6f8be
SIENTIAPDE-1163 SIENTIAPDE-1163 Update test_redis.py to mock existing data in redis_activity.get for improved error handling in group and hold data tests.
vitor-aignosi
2025-07-17 14:30:34 -03:00
839be18644
SIENTIAPDE-1163 Update .gitignore to include .coverage and git_log for better file management
vitor-aignosi
2025-07-17 12:33:27 -03:00
b0d0478b20
SIENTIAPDE-1163 Add PR shortcut command to README for easier log generation
vitor-aignosi
2025-07-17 12:32:38 -03:00
939acb5354
SIENTIAPDE-1163 Update image tag in values.yaml from 0.2.3 to 0.2.4 for version increment.
vitor-aignosi
2025-07-17 12:31:08 -03:00
9329ae206c
SIENTIAPDE-1163 Update sientia-dataops-library dependency version from 1.3.0 to 1.3.3 in requirements.txt and change GITHUB_BRANCH in values.yaml to SIENTIAPDE-1163-alterar-dinamica-de-notificacoes-para-usar-o-mongodb-ao-inves-do-kafka for improved notification handling.
vitor-aignosi
2025-07-17 11:00:52 -03:00
03f5518cd1
SIENTIAPDE-1163 Update sientia-dataops-library dependency version from 1.2.1 to 1.3.0 in requirements.txt
vitor-aignosi
2025-07-16 09:23:02 -03:00
Matheus Demoner