SIENTIAPDE-1005

Implement workflows for fake data generation, scouter processing, and core scouter operations

- Added `FakeData` workflow to generate random data and send it to a Kafka topic.
- Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow.
- Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL.
- Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling.
- Enhanced Redis and Postgres activities with robust testing for data handling and error notifications.
- Introduced quality filters for data validation and implemented tests to verify their functionality.
This commit is contained in:
vitor-aignosi
2025-05-15 16:53:24 -03:00
parent 4e579dd5bd
commit b203b7d22c
29 changed files with 2070 additions and 49 deletions

20
.env Normal file
View File

@@ -0,0 +1,20 @@
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_USER=sientia
POSTGRES_PASSWORD=sientia
POSTGRES_DB=sientia
POSTGRES_MIN_CONNECTIONS=5
POSTGRES_MAX_CONNECTIONS=20
KAFKA_BOOTSTRAP_SERVERS=kafka:29092
KAFKA_POLLING_TIME=1000
REDIS_HOST=redis
REDIS_PORT=6379
TEMPORAL_HOST=host.docker.internal:7233
TEMPORAL_NAMESPACE=default
LOG_LEVEL=INFO
PROJECT_NAME=scouter