SIENTIAPDE-1005

Implement workflows for fake data generation, scouter processing, and core scouter operations

- Added `FakeData` workflow to generate random data and send it to a Kafka topic.
- Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow.
- Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL.
- Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling.
- Enhanced Redis and Postgres activities with robust testing for data handling and error notifications.
- Introduced quality filters for data validation and implemented tests to verify their functionality.
This commit is contained in:
vitor-aignosi
2025-05-15 16:53:24 -03:00
parent 4e579dd5bd
commit b203b7d22c
29 changed files with 2070 additions and 49 deletions

View File

@@ -0,0 +1,38 @@
from unittest.mock import AsyncMock, patch, ANY
from pytest import fixture, mark
from scouter.workflow.scouter import Scouter
from scouter.activities.activities import Activities
@fixture
def scouter():
return Scouter()
@mark.asyncio
@patch('scouter.workflow.scouter.workflow', new_callable=AsyncMock)
async def test_scouter_workflow(mock_workflow, scouter):
mock_workflow.execute_activity_method.return_value = 'test_data'
await scouter.run(
input_data={
'topic': 'test_topic'
}
)
mock_workflow.execute_activity_method.assert_called_once_with(
Activities.load_from_kafka,
{
'topic': 'test_topic'
},
retry_policy=ANY,
start_to_close_timeout=ANY
)
mock_workflow.execute_child_workflow.assert_called_once_with(
'core_scouter',
{
'topic': 'test_topic',
'data': 'test_data'
}
)