b203b7d22c
Implement workflows for fake data generation, scouter processing, and core scouter operations - Added `FakeData` workflow to generate random data and send it to a Kafka topic. - Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow. - Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL. - Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling. - Enhanced Redis and Postgres activities with robust testing for data handling and error notifications. - Introduced quality filters for data validation and implemented tests to verify their functionality.
31 lines
872 B
Docker
31 lines
872 B
Docker
# Use uma imagem base Python
|
|
FROM python:3.11-slim
|
|
|
|
# Instale git e outras dependências do sistema
|
|
RUN apt-get update && apt-get install -y git \
|
|
&& apt-get install -y build-essential python3-dev \
|
|
&& apt-get install -y vim \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Defina o diretório de trabalho
|
|
WORKDIR /app
|
|
|
|
# Copie os arquivos do projeto
|
|
COPY . /app
|
|
|
|
RUN pip install --upgrade pip setuptools wheel
|
|
|
|
# Install the required packages
|
|
# Add github to known hosts
|
|
# This is needed for SSH to work
|
|
# The SSH key will NOT remain in the image
|
|
# IMPORTANT: this block requires BuildKit
|
|
# and the --ssh flag during docker build
|
|
RUN --mount=type=ssh \
|
|
mkdir -p ~/.ssh && \
|
|
ssh-keyscan github.com >> ~/.ssh/known_hosts && \
|
|
pip install --no-cache-dir -r requirements.txt
|
|
|
|
|
|
# Defina o comando para executar o worker
|
|
CMD ["python", "-m", "scouter.worker.worker"] |