SIENTIAPDE-1005

Implement workflows for fake data generation, scouter processing, and core scouter operations

- Added `FakeData` workflow to generate random data and send it to a Kafka topic.
- Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow.
- Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL.
- Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling.
- Enhanced Redis and Postgres activities with robust testing for data handling and error notifications.
- Introduced quality filters for data validation and implemented tests to verify their functionality.
This commit is contained in:
vitor-aignosi
2025-05-15 16:53:24 -03:00
parent 4e579dd5bd
commit b203b7d22c
29 changed files with 2070 additions and 49 deletions

31
Dockerfile Normal file
View File

@@ -0,0 +1,31 @@
# Use uma imagem base Python
FROM python:3.11-slim
# Instale git e outras dependências do sistema
RUN apt-get update && apt-get install -y git \
&& apt-get install -y build-essential python3-dev \
&& apt-get install -y vim \
&& rm -rf /var/lib/apt/lists/*
# Defina o diretório de trabalho
WORKDIR /app
# Copie os arquivos do projeto
COPY . /app
RUN pip install --upgrade pip setuptools wheel
# Install the required packages
# Add github to known hosts
# This is needed for SSH to work
# The SSH key will NOT remain in the image
# IMPORTANT: this block requires BuildKit
# and the --ssh flag during docker build
RUN --mount=type=ssh \
mkdir -p ~/.ssh && \
ssh-keyscan github.com >> ~/.ssh/known_hosts && \
pip install --no-cache-dir -r requirements.txt
# Defina o comando para executar o worker
CMD ["python", "-m", "scouter.worker.worker"]