Commit Graph

9 Commits

Author SHA1 Message Date
vitor-aignosi
51f4eb1435 SIENTIAPDE-1110
Add new activities to worker.py for data management: load_latest_data, get_last_data_timestamp, and put_last_data_timestamp.
2025-07-01 15:16:55 -03:00
vitor-aignosi
97d66ed6cd SIENTIAPDE-1110
Add MongoDB configuration to values.yaml and integrate MongoDB support in worker.py
2025-07-01 15:15:18 -03:00
vitor-aignosi
74a2d9aa3e SIENTIAPDE-1110
Update GITHUB_BRANCH in values.yaml to SIENTIAPDE-1110-criar-testes-e-2-e and remove prepare_activity from worker.py
2025-06-27 09:34:58 -03:00
vitor-aignosi
8cf3273714 SIENTIAPDE-1082
Implement shutdown methods for Activities and Kafka; enhance error handling in worker main function
2025-06-09 15:09:53 -03:00
vitor-aignosi
7dbb9a29ea SIENTIAPDE-1094
Update environment configuration and refactor activity imports

- Changed Kafka, Redis, and Temporal host configurations to use localhost.
- Updated the version reference for the sientia-dataops-library in requirements.txt.
- Refactored import paths for activities to align with new module structure.
- Removed unused base.py and postgres.py files.
- Updated logger and policies imports to reflect new module locations.
- Adjusted values.yaml for branch and log level settings.
2025-06-09 09:43:06 -03:00
vitor-aignosi
3050b66737 SIENTIAPDE-1005
Prepare activity fix
2025-05-19 11:51:49 -03:00
vitor-aignosi
e252ca7962 SIENTIAPDE-1005
Full coverage

Refactor code structure and improve logging configuration; update tests for activities and connectors
2025-05-16 12:58:36 -03:00
vitor-aignosi
b203b7d22c SIENTIAPDE-1005
Implement workflows for fake data generation, scouter processing, and core scouter operations

- Added `FakeData` workflow to generate random data and send it to a Kafka topic.
- Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow.
- Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL.
- Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling.
- Enhanced Redis and Postgres activities with robust testing for data handling and error notifications.
- Introduced quality filters for data validation and implemented tests to verify their functionality.
2025-05-15 16:53:24 -03:00
vitor-aignosi
02bb052605 File structure 2025-05-13 12:09:26 -03:00