SIENTIAPDE-1005
Implement workflows for fake data generation, scouter processing, and core scouter operations - Added `FakeData` workflow to generate random data and send it to a Kafka topic. - Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow. - Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL. - Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling. - Enhanced Redis and Postgres activities with robust testing for data handling and error notifications. - Introduced quality filters for data validation and implemented tests to verify their functionality.
This commit is contained in:
31
Dockerfile
Normal file
31
Dockerfile
Normal file
@@ -0,0 +1,31 @@
|
||||
# Use uma imagem base Python
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Instale git e outras dependências do sistema
|
||||
RUN apt-get update && apt-get install -y git \
|
||||
&& apt-get install -y build-essential python3-dev \
|
||||
&& apt-get install -y vim \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Defina o diretório de trabalho
|
||||
WORKDIR /app
|
||||
|
||||
# Copie os arquivos do projeto
|
||||
COPY . /app
|
||||
|
||||
RUN pip install --upgrade pip setuptools wheel
|
||||
|
||||
# Install the required packages
|
||||
# Add github to known hosts
|
||||
# This is needed for SSH to work
|
||||
# The SSH key will NOT remain in the image
|
||||
# IMPORTANT: this block requires BuildKit
|
||||
# and the --ssh flag during docker build
|
||||
RUN --mount=type=ssh \
|
||||
mkdir -p ~/.ssh && \
|
||||
ssh-keyscan github.com >> ~/.ssh/known_hosts && \
|
||||
pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
|
||||
# Defina o comando para executar o worker
|
||||
CMD ["python", "-m", "scouter.worker.worker"]
|
||||
Reference in New Issue
Block a user