e879107053
SIENTIAPDE-1441 SIENTIAPDE-1442: Enhance Temporal worker configuration by adding environment variables for concurrency limits and poller behavior settings in values.yaml and worker.py for optimized performance.
vitor-aignosi2025-12-16 10:10:42 -03:00
f3833a0f98
SIENTIAPDE-1441 Remove encrypt.py and test.ipynb files; update values.yaml to set image tag to 1.0.0 and change GITHUB_BRANCH to release/SIENTIAPDE-1441 for version consistency.
vitor-aignosi2025-12-16 09:48:28 -03:00
2bc49637a1
SIENTIAPDE-1273 Update image tag in values.yaml from 0.5.0 to 0.5.2 for version consistency.
vitor-aignosi2025-11-18 11:42:21 -03:00
fc4ec35298
SIENTIAPDE-1273 Update GITHUB_BRANCH in values.yaml to fix/SIENTIAPDE-1273 for issue resolution.
vitor-aignosi2025-11-17 16:53:32 -03:00
6701a25c7b
SIENTIAPDE-1273 Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1273, increment Helm chart version to 0.6.0, and modify metrics registration in Gates class to skip None values. Update test cases to reflect changes in tag values.
vitor-aignosi2025-11-14 12:04:16 -03:00
5db04c7e5b
Merge pull request #24 from Aignosi/feature/SIENTIAPDE-1325-adicionar-metricas-especificas-de-operacoes-externas
vitor-aignosi2025-11-10 10:12:16 -03:00
1b7391e8f1
SIENTIAPDE-1325 Enhance Redis activity by formatting set and get operations for improved readability and consistency. Update tests to include metadata in assertions for better verification of Redis interactions.
vitor-aignosi2025-11-10 08:39:19 -03:00
3e8ba6c758
SIENTIAPDE-1325 Update release workflow to trigger only on merged pull requests, ensuring proper release management.
vitor-aignosi2025-11-10 08:23:01 -03:00
082ae682ee
SIENTIAPDE-1325 Update requirements to use sientia-dataops-library version 1.5.3 and enhance Redis activity logging by including metadata in get and set operations.
vitor-aignosi2025-11-06 15:19:55 -03:00
d4fbb69b26
SIENTIAPDE-1325 Update requirements to use the remote sientia-dataops-library repository and increment image tag in values.yaml from 0.4.9 to 0.5.0 for version consistency.
vitor-aignosi2025-11-05 09:28:17 -03:00
32882d0b6f
SIENTIAPDE-1325 Update metric labels from 'pipeline_name' to 'workflow_name' in metrics and related classes, ensuring consistency across the codebase and tests.
vitor-aignosi2025-11-04 08:49:38 -03:00
4351289111
SIENTIAPDE-1325 Refactor notification methods in Gates, MongoDB, and Redis classes to use asynchronous send_notification_async. Update related tests to ensure proper async handling and verification of notifications.
vitor-aignosi2025-11-03 14:57:44 -03:00
8350a3a0c3
SIENTIAPDE-1325 Refactor initialization of Activities, Gates, and MongoDB classes for improved readability by using multi-line argument formatting. Update related tests to match the new initialization style.
vitor-aignosi2025-10-30 16:50:39 -03:00
d4a74ec5eb
SIENTIAPDE-1325 Update environment configuration and refactor activities to include metrics controller. Remove Kafka settings and adjust Redis and MongoDB initialization. Update tests to reflect changes in initialization and metrics tracking.
vitor-aignosi2025-10-30 16:49:55 -03:00
0c16075823
SIENTIAPDE-1325 Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1325 for specific external operation metrics. Refactor shutdown methods in Activities class to ensure proper closure of MongoDB, Redis, and Gates connections. Adjust tests to reflect these changes and enhance repository pattern implementation for MongoDB and Redis activities.
vitor-aignosi2025-10-29 16:58:22 -03:00
10b3310677
SIENTIAPDE-1325 Refactor MongoDB, Redis, and Gates classes to extend SientiaMonitoring instead of BaseActivity. Update requirements to point to local dataops library path. Implement repository pattern for MongoDB and Redis operations, enhancing code organization and maintainability.
vitor-aignosi2025-10-29 16:01:54 -03:00
5dc2971d8c
SIENTIAPDE-1314 SIENTIAPDE-1314: Update workflow name from 'sub_core_scouter' to 'subworkflow.core_scouter' in scouter.py and core_scouter.py, and adjust test references accordingly in test_scouter.py for consistency.
vitor-aignosi2025-10-27 10:16:57 -03:00
6af45c7476
SIENTIAPDE-1314 SIENTIAPDE-1314: Rename core_scouter workflow to sub_core_scouter and update references in scouter.py and test_scouter.py for consistency.
vitor-aignosi2025-10-27 10:11:07 -03:00
25838c26a6
SIENTIAPDE-1314 Update GITHUB_BRANCH in values.yaml to fix/SIENTIAPDE-1314 for monitoring layer adjustments
vitor-aignosi2025-10-24 10:14:26 -03:00
d1521838c5
SIENTIAPDE-1314 SIENTIAPDE-1314: Update type annotations in encrypt.py for improved clarity and type safety. Added type hint for file_tree variable and included type ignore for yaml import.
vitor-aignosi2025-10-23 16:15:29 -03:00
05fa808bfd
SIENTIAPDE-1314 SIENTIAPDE-1314: Remove redundant blank line in Redis class to enhance code clarity.
vitor-aignosi2025-10-23 16:10:36 -03:00
fb1440b0d7
SIENTIAPDE-1314 Remove unnecessary blank line in Redis class definition to improve code readability.
vitor-aignosi2025-10-23 16:08:44 -03:00
f0ca06b4aa
SIENTIAPDE-1314 Implement tag value registration in Gates activity and update related tests. Enhanced the write_metrics method to process and register tag values, improving metrics tracking. Adjusted tests to validate new functionality.
vitor-aignosi2025-10-23 16:02:38 -03:00
c9136ca537
Merge pull request #22 from Aignosi/feature/SIENTIAPDE-1327-controle-de-releases
Bruno Domingues
2025-10-23 10:45:15 -03:00
b063f973cd
SIENTIAPDE-1327 Update requirements and simplify CI workflow by integrating quality gate from template. Bump sientia-dataops-library version to 1.4.7 and streamline quality-gate.yml to use a shared workflow configuration.
vitor-aignosi2025-10-23 09:04:28 -03:00
507ee8cc18
SIENTIAPDE-1312 Update image tag to 0.4.9 and change GITHUB_BRANCH to SIENTIAPDE-1312-melhorias-e-correcoes-nas-pipelines-de-dados in values.yaml
vitor-aignosi2025-10-21 14:40:07 -03:00
9cc55ea5e7
SIENTIAPDE-1312 Update Redis and CoreScouter to incorporate 'fill_missing_tags' parameter in data processing. Adjusted related tests to ensure proper handling of missing tags, enhancing overall functionality and consistency across workflows.
vitor-aignosi2025-10-21 09:14:41 -03:00
bd6474a975
SIENTIAPDE-1312 SIENTIAPDE-1318: Add 'fill_missing_tags' parameter to CoreScouter for enhanced tag handling in data processing. This update allows for the specification of missing tags during workflow execution.
vitor-aignosi2025-10-21 09:01:08 -03:00
4bd29ae0d7
SIENTIAPDE-1318 SIENTIAPDE-1318: Add support for filling missing tags in Redis data processing. Enhanced the Redis class to include a new parameter for handling missing tags, ensuring that absent tags are filled with None in the data package.
vitor-aignosi2025-10-20 16:24:36 -03:00
97eb5bc904
SIENTIAPDE-1316 Update .gitignore and refactor metrics.py, activities.py, and gates.py for improved clarity and consistency. Added coverage.xml and cache directories to .gitignore. Standardized string formatting and parameter handling in metrics and activities classes, enhancing code readability. Removed the deprecated faker.py file and adjusted related tests accordingly.
vitor-aignosi2025-10-16 13:31:12 -03:00
8bdbf049b8
SIENTIAPDE-1316 SIENTIAPDE-1084: Refactor apply_aggregation method in Gates class for improved performance and clarity. Changed parameter name from 'group' to 'values', optimized aggregation logic, and enhanced error handling for invalid functions. Streamlined data processing by reducing unnecessary operations and added batch debug logging.
vitor-aignosi2025-10-16 13:06:44 -03:00
205284d9f4
Merge pull request #18 from Aignosi/SIENTIAPDE-1084-ajustar-documentacao
Bruno Domingues
2025-09-03 13:33:41 +00:00
199b1deacb
SIENTIAPDE-1084 Update test_activities.py to change assertions for mock_postgres_close and mock_mongodb_close from assert_called_once to assert_called, allowing for more flexible verification of calls during shutdown tests.
vitor-aignosi2025-09-01 16:07:42 -03:00
1a16a17267
SIENTIAPDE-1084 Remove quality filters from the CoreScouter Workflow flowchart in README.md to enhance clarity and focus on the main data processing steps.
vitor-aignosi2025-08-29 16:57:27 -03:00
4c929fe866
SIENTIAPDE-1084 Remove outdated key components section from README.md to streamline the documentation and focus on the CoreScouter Workflow.
vitor-aignosi2025-08-29 16:55:42 -03:00
da15f2d40d
SIENTIAPDE-1084 Update README.md to modify the Scouter Workflow flowchart, replacing the icon for the core_scouter step with a refresh symbol for better visual representation of data processing continuity.
vitor-aignosi2025-08-29 16:55:21 -03:00
864bf0af96
SIENTIAPDE-1084 Update README.md to correct the flowchart syntax for the Scouter Workflow Architecture, changing from 'flowchart-elk' to 'flowchart' to ensure proper rendering and maintain clarity in visualizing data processing steps.
vitor-aignosi2025-08-29 16:43:05 -03:00
abdec12635
SIENTIAPDE-1084 Update README.md to modify the Scouter Workflow Architecture flowchart, changing the layout to 'flowchart-elk' for improved clarity in visualizing data processing steps and interactions.
vitor-aignosi2025-08-29 16:42:33 -03:00
e573923bdd
SIENTIAPDE-1084 Refactor README.md to update the Scouter Workflow Architecture flowchart, transitioning to a left-to-right layout for improved readability and clarity in depicting data processing steps and service interactions.
vitor-aignosi2025-08-29 16:41:09 -03:00
791d8b8be2
SIENTIAPDE-1084 Update README.md to revise the Scouter Workflow Architecture flowchart, replacing outdated steps with a new data processing sequence that includes data quality checks, aggregation, and metrics writing, enhancing clarity on system operations and service interactions.
vitor-aignosi2025-08-29 16:39:31 -03:00
6c22b00d5f
SIENTIAPDE-1084 Update README.md to enhance the architecture diagrams for both Scouter and FakeData workflows, improving clarity and structure by adopting a consistent flowchart format and detailing service interactions.
vitor-aignosi2025-08-29 16:37:34 -03:00
79dc20e993
SIENTIAPDE-1084 Update README.md to modify the Scouter Workflow Architecture flowchart, enhancing clarity by restructuring the diagram layout and correcting the connections to Redis and MongoDB components.
vitor-aignosi2025-08-29 16:31:02 -03:00
f5c4d38a7a
SIENTIAPDE-1084 Refactor README.md to streamline the Scouter Workflow Architecture section by consolidating the flowchart into a single line format for improved readability and clarity.
vitor-aignosi2025-08-29 16:28:49 -03:00
cbad5df8d2
SIENTIAPDE-1084 Update README.md to include a new section on Scouter Workflow Architecture, detailing the incremental data processing steps and key components, while removing outdated system architecture diagrams for improved clarity and relevance.
vitor-aignosi2025-08-29 16:27:21 -03:00
fe6133f31a
SIENTIAPDE-1084 Enhance README.md by adding detailed system architecture and data processing flow diagrams using Mermaid syntax, improving clarity on the Scouter system's components and interactions.
vitor-aignosi2025-08-29 16:20:07 -03:00
beb06d257f
SIENTIAPDE-1084 Update README.md to change the section title from "Temporal input parameters" to "MongoDB pipeline configuration" for improved accuracy in documentation.
vitor-aignosi2025-08-29 16:04:07 -03:00
cf1ce1a38d
SIENTIAPDE-1084 Enhance README.md by adding a section for Scouter Workflow Input Parameters to improve documentation clarity on system configuration.
vitor-aignosi2025-08-29 16:02:34 -03:00
9dc3a4c55a
SIENTIAPDE-1084 Update README.md to reflect new workflow configuration parameters, including temporal input settings, tag definitions, and filter policies for enhanced clarity on system capabilities.
vitor-aignosi2025-08-29 15:19:32 -03:00
88576252af
SIENTIAPDE-1084 Update .gitignore to include git_log and .env files for better environment management.
vitor-aignosi2025-08-29 11:57:05 -03:00
a973da9d60
SIENTIAPDE-1084 Remove deprecated files and configurations, including .env, Dockerfile, docker-compose.yml, and client-schedule.py. Update README.md to reflect new architecture and features, enhancing clarity on system capabilities and workflows. Adjust values.yaml for image tag and replica count, and improve code documentation across various modules for better maintainability.
vitor-aignosi2025-08-29 11:56:45 -03:00
00d25bdefc
Merge pull request #17 from Aignosi/SIENTIAPDE-1193-conferir-como-a-escrita-de-datetime-ocorre-no-temporal
Bruno Domingues
2025-08-26 13:39:06 +00:00
730c9e9ff9
Merge branch 'main' into SIENTIAPDE-1193-conferir-como-a-escrita-de-datetime-ocorre-no-temporal
vitor-aignosi2025-08-25 10:46:36 -03:00
719e3917ab
SIENTIAPDE-1193 Refactor Redis key format in tests for last data timestamp to use colons, enhancing readability and consistency with recent changes.
vitor-aignosi2025-08-25 09:03:03 -03:00
5c01d024a8
SIENTIAPDE-1193 Update image tag in values.yaml from 0.4.3 to 0.4.4 for version increment.
vitor-aignosi2025-08-25 08:28:12 -03:00
18f7ab6723
SIENTIAPDE-1193 Update replica count in values.yaml from 1 to 3 to scale application deployment.
vitor-aignosi2025-08-22 15:46:14 -03:00
826b559489
SIENTIAPDE-1193 Refactor CoreScouter workflow test to use updated DATETIME_FORMAT_WITH_TZ, ensuring consistency with recent changes in timestamp handling.
vitor-aignosi2025-08-22 15:44:13 -03:00
7bbfa4dbac
SIENTIAPDE-1193 Refactor timestamp handling in CoreScouter workflow by updating to DATETIME_FORMAT_WITH_TZ, ensuring consistency with recent changes in datetime management.
vitor-aignosi2025-08-22 15:33:41 -03:00
54c3dd25e9
SIENTIAPDE-1193 Update replica count in values.yaml to 1 and enhance timestamp handling in MongoDB activity by setting timezone to UTC for inserted_at field.
vitor-aignosi2025-08-22 15:30:47 -03:00
39ba82b539
SIENTIAPDE-1193 Update image tag to 0.4.3 in values.yaml and refactor Redis key format for last data timestamp to use colons for improved readability.
vitor-aignosi2025-08-22 15:13:04 -03:00
96a28e91e5
SIENTIAPDE-1193 Enhance timestamp handling in Redis and CoreScouter workflows by updating to DATETIME_FORMAT_MS_WITH_TZ. Modify tests to reflect new timestamp format, ensuring consistency across data structures and improving overall datetime management.
vitor-aignosi2025-08-22 14:05:41 -03:00
0b337d00f9
SIENTIAPDE-1193 Refactor Redis activity to utilize the new 'now' function for timestamp generation, ensuring consistency with updated datetime handling. This change replaces the direct use of 'datetime.now()' with 'now()' from the temporal constants.
vitor-aignosi2025-08-22 11:24:26 -03:00
0e0603d2bc
SIENTIAPDE-1193 Update sientia-dataops-library version to 1.4.3 and refactor timestamp handling in MongoDB and Redis activities to use DATETIME_FORMAT constants for improved consistency. Enhance unit tests to reflect these changes.
vitor-aignosi2025-08-22 11:12:37 -03:00
eabc642feb
SIENTIAPDE-1193 Update dependencies in requirements.txt and change GITHUB_BRANCH in values.yaml to reflect new datetime handling task.
vitor-aignosi2025-08-22 08:02:13 -03:00
fad10bc0d0
SIENTIAPDE-1193 Update requirements and core scouter workflow: comment out old dataops library dependency, change default namespace in worker.py, and enhance timestamp handling in core_scouter.py with DATETIME_FORMAT.
vitor-aignosi2025-08-21 12:16:06 -03:00
799aa5af3e
SIENTIAPDE-1199 Refactor logging in worker.py to use custom_info and custom_error methods, enhancing log metadata with pod and workflow details for improved observability.
vitor-aignosi2025-08-21 11:35:26 -03:00
76d96c2c13
Merge branch 'main' into SIENTIAPDE-1199-revisar-e-testar-observabilidade
vitor-aignosi2025-08-21 08:36:38 -03:00
2a76f2e3c6
SIENTIAPDE-1199 Enhance logging in Gates, MongoDB, and Redis activities by replacing debug statements with info level logs, improving observability of data processing steps. Update worker configuration to adjust concurrency settings and enable autoscaling for task polling.
vitor-aignosi2025-08-20 12:39:43 -03:00
3bbc2c4993
SIENTIAPDE-1199 Update dependencies and improve observability by changing logger imports. Bump sientia-dataops-library version to 1.4.1 and update image tag to 0.4.2 in values.yaml.
vitor-aignosi2025-08-19 16:29:30 -03:00
552f58ca86
SIENTIAPDE-1169 SIENTIAPDE-1169 Update Redis and core scouter tests to include model_tags in data structures, enhancing data organization and consistency across test cases.
vitor-aignosi2025-08-19 09:29:46 -03:00
2c0f5a67e6
Merge branch 'main' into SIENTIAPDE-1169-pensar-e-projetar-testes-de-breakdown-e-performance
vitor-aignosi2025-08-18 15:20:23 -03:00
9480495155
SIENTIAPDE-1169 Enable SDK metrics service in values.yaml and update worker.py to initialize SDK metrics server with Prometheus configuration, enhancing observability of the application.
vitor-aignosi2025-08-18 13:45:20 -03:00
4bbd937b7f
SIENTIAPDE-1169 Enhance Redis activity logging by adding debug statements for tag management and metrics registration, improving traceability of data processing.
vitor-aignosi2025-08-14 16:33:39 -03:00
9303158a9d
SIENTIAPDE-1169 Update image tag to 0.4.0 in values.yaml, modify GITHUB_BRANCH for SIENTIAPDE-1169, and enhance Redis activity to handle model tags for improved data retention.
vitor-aignosi2025-08-14 16:15:20 -03:00
77b65bf740
Merge pull request #14 from Aignosi/SIENTIAPDE-1174-mapear-e-implementar-metricas-a-serem-criadas
Bruno Domingues
2025-08-13 16:34:49 -03:00
ba994f9bb4
Merge branch 'main' into SIENTIAPDE-1174-mapear-e-implementar-metricas-a-serem-criadas
vitor-aignosi2025-08-05 14:04:46 -03:00
c6fca2a8c2
SIENTIAPDE-1174 Update image tag to 0.3.2 in values.yaml for version consistency in deployment.
vitor-aignosi2025-08-04 15:26:21 -03:00
d4718c7d95
SIENTIAPDE-1174 Update Helm upgrade command in values.yaml to use version 0.4.0, ensuring consistency in deployment instructions.
vitor-aignosi2025-08-04 10:04:01 -03:00
07c54e9edd
Merge pull request #13 from Aignosi/SIENTIAPDE-1172-criar-pipeline-de-alertas-orquestrador
Bruno Domingues
2025-08-04 10:02:39 -03:00
25f9d89317
SIENTIAPDE-1174 Update image tag to 0.3.1 in values.yaml and modify write_metrics method signature in Gates activity for improved type clarity.
vitor-aignosi2025-08-01 08:36:08 -03:00
40c57ed23d
SIENTIAPDE-1174 Implement async test for write_metrics method in Gates activity, enhancing test coverage for metric logging functionality.
vitor-aignosi2025-08-01 07:40:06 -03:00
fd740ca451
SIENTIAPDE-1174 Update Gates activity to use labels in LABORIOUS_DATA_WRITTEN_COUNT metric, improving clarity in metric tracking.
vitor-aignosi2025-07-31 16:34:55 -03:00
7ea9c1ce65
SIENTIAPDE-1174 Update Gates and Redis activities to use 'workflow_name' instead of 'pipeline_name' in metric tracking, ensuring consistency across the codebase.
vitor-aignosi2025-07-31 16:31:38 -03:00
b9c004827f
SIENTIAPDE-1174 Update Redis activity to use metadata for model and pipeline names, improving consistency in metric tracking.
vitor-aignosi2025-07-31 16:27:18 -03:00
a8222359cc
SIENTIAPDE-1174 Refactor Scouter class to remove pod_id reference, enhancing code clarity and consistency in data handling.
vitor-aignosi2025-07-31 16:25:02 -03:00
205810e882
SIENTIAPDE-1174 Update requirements for sientia-dataops-library to version 1.3.7, add ServiceMonitor configuration for Prometheus in values.yaml, and refactor pod_id usage in Gates and Redis activities for improved consistency in metric tracking.
vitor-aignosi2025-07-31 15:45:18 -03:00
c69ae5df03
SIENTIAPDE-1174 Remove unused getenv import from scouter.py to enhance code clarity and maintainability.
vitor-aignosi2025-07-31 14:59:12 -03:00
5d3eb74022
SIENTIAPDE-1174 SIENTIAPDE-1174 Add pod_id initialization in Activities class and update Scouter to reference it for improved consistency in environment variable handling.
vitor-aignosi2025-07-31 14:57:22 -03:00
ce20d86328
SIENTIAPDE-1174 Remove unused metrics import from core_scouter.py to streamline code and improve clarity.
vitor-aignosi2025-07-31 14:48:00 -03:00
b4827935a7
SIENTIAPDE-1174 Integrate write_metrics activity into CoreScouter for enhanced metric logging and update worker.py to include write_metrics in workflow tasks.
vitor-aignosi2025-07-31 14:43:34 -03:00
1c78cfb558
SIENTIAPDE-1174 Add write_metrics activity to Gates for database metric logging and import metrics module
vitor-aignosi2025-07-31 14:39:58 -03:00
7925b8a2ae
SIENTIAPDE-1174 Update replica count in values.yaml, add application status metrics in metrics.py, and implement Prometheus server in worker.py for enhanced observability.
vitor-aignosi2025-07-31 13:01:39 -03:00
vitor-aignosi
Matheus Demoner