Update sientia-dataops-library version to 1.4.3 and refactor timestamp handling in MongoDB and Redis activities to use DATETIME_FORMAT constants for improved consistency. Enhance unit tests to reflect these changes.
Enhance logging in Gates, MongoDB, and Redis activities by replacing debug statements with info level logs, improving observability of data processing steps. Update worker configuration to adjust concurrency settings and enable autoscaling for task polling.
Update dependencies and improve observability by changing logger imports. Bump sientia-dataops-library version to 1.4.1 and update image tag to 0.4.2 in values.yaml.
Update image tag to 0.4.0 in values.yaml, modify GITHUB_BRANCH for SIENTIAPDE-1169, and enhance Redis activity to handle model tags for improved data retention.
Update requirements for sientia-dataops-library to version 1.3.7, add ServiceMonitor configuration for Prometheus in values.yaml, and refactor pod_id usage in Gates and Redis activities for improved consistency in metric tracking.
SIENTIAPDE-1174 Add pod_id initialization in Activities class and update Scouter to reference it for improved consistency in environment variable handling.
Implement store_data_package method in Redis activity for debug data storage. Update CoreScouter workflow to conditionally call store_data_package based on debug flag. Enhance tests for store_data_package functionality and error handling.
Enhance Gates and Redis activities by adding metadata parameter to apply_aggregation and notification methods. Refactor notification handling to use send_notification for improved consistency. Update tests to reflect changes in notification method calls and ensure proper functionality with new metadata integration.
Refactor Activities class to remove Kafka and Druid dependencies, simplifying initialization. Update values.yaml to set replica count to 1 for reduced resource usage. Adjust Redis activity to set TTL to None for better data retention. Remove unused Kafka and Druid activity files and their associated tests, streamlining the codebase.
Update Druid configuration in values.yaml to change port from 8081 to 8888. Refactor pydruid.py to use druid_engine for SQLAlchemy connections, enhancing clarity and consistency in data loading operations.
Update Druid activity to handle None values for last data timestamp, ensuring default timestamp is set to epoch start. Enhance data loading debug logging for improved traceability.
Enhance Druid activity by adding configuration for Druid host and port in values.yaml. Refactor data loading logic to utilize SQLAlchemy for querying Druid, improving query efficiency and readability. Update timestamp handling to streamline data retrieval process.
Update Druid activity to handle None values for last data timestamp, ensuring robust date parsing. Bump image tag in values.yaml to 0.2.3 for version consistency.
Integrate Druid activity into the Activities class, adding support for Druid configuration and initialization. Update worker and connectors configuration to accommodate Druid, enhancing data processing capabilities.
Update MongoDB activity to format 'inserted_at' timestamps as strings in the format 'YYYY-MM-DD HH:MM:SS.ssssss', improving readability of timestamp data.
Refactor Kafka activity to improve message consumption by using getmany for batch retrieval, enhancing performance and simplifying message handling logic.
Refactor Kafka activity to use instance-level kafka_connector for improved message polling and management, replacing local variable usage with class attribute methods.
