Implement store_data_package method in Redis activity for debug data storage. Update CoreScouter workflow to conditionally call store_data_package based on debug flag. Enhance tests for store_data_package functionality and error handling.
Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
Update environment configuration and refactor activity imports
- Changed Kafka, Redis, and Temporal host configurations to use localhost.
- Updated the version reference for the sientia-dataops-library in requirements.txt.
- Refactored import paths for activities to align with new module structure.
- Removed unused base.py and postgres.py files.
- Updated logger and policies imports to reflect new module locations.
- Adjusted values.yaml for branch and log level settings.
Implement workflows for fake data generation, scouter processing, and core scouter operations
- Added `FakeData` workflow to generate random data and send it to a Kafka topic.
- Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow.
- Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL.
- Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling.
- Enhanced Redis and Postgres activities with robust testing for data handling and error notifications.
- Introduced quality filters for data validation and implemented tests to verify their functionality.
