Enhance API class to format timestamps correctly and improve logging. Update test cases to include 'end_time' parameter for better data retrieval flexibility.
Refactor API class to remove unnecessary blank line and improve code readability. Update test cases to rename 'timeout' parameter to 'request_timeout' for consistency.
Refactor metrics and imports across multiple files
- Removed unused import of CORE_LABELS in metrics.py.
- Updated import statements in api.py for consistency.
- Cleaned up import order in worker.py for better readability.
- Added missing newline at the end of metrics.py.
- Adjusted formatting in core_scouter.py to ensure proper syntax.
- Removed redundant 'on_conflict' and 'unique_columns' keys from test cases in test_core_scouter.py for clarity.
Update pytest_asyncio fixture scopes in conftest.py for improved test isolation and add asyncio_default_fixture_loop_scope in pyproject.toml. Remove outdated scenarios from scenarios.md and delete unused test files for cleaner codebase.
Refactor PIWebAPIClient and Update Configuration Imports
- Removed outdated PostgreSQL, Redis, MongoDB, and API configuration functions from connectors_config.py.
- Deleted the pi_web_api_client.py file as part of the refactor.
- Updated imports in worker.py and api.py to use the new repository structure.
- Cleaned up scenarios.md by removing obsolete scenarios related to unique constraint violations.
- Commented out the specific version of the sientia-dataops-library in requirements.txt for flexibility.
Invert startTime and endTime parameters in PIWebAPIClient to ensure descending order results. Update tests to verify parameter handling and use pytest.approx for floating-point comparisons.
Update test_api.py and test_core_scouter.py to enhance timestamp validation and add new test cases
- Updated timestamp assertions in test_get_tag_values_success to reflect the correct values.
- Added new test cases in test_core_scouter.py to handle scenarios with zero affected rows and without debug data package, ensuring proper workflow execution and activity method assertions.
Refactor imports and enhance timestamp handling in API and test files
- Removed redundant import statements in api.py and prepare_worker.py for cleaner code.
- Updated timestamp format in test cases to include timezone information for consistency.
- Adjusted test configurations to align with new API structure for improved accuracy in data handling.
Enhance Activities and API Integration
- Updated Activities class to include API operations for external data ingestion.
- Added API configuration builder to connectors_config.py for environment variable management.
- Integrated API configuration into worker setup.
- Expanded unit tests to cover new API functionality and configuration handling.
- Updated requirements.txt to include pycurl and prometheus-client for enhanced metrics support.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1273, increment Helm chart version to 0.6.0, and modify metrics registration in Gates class to skip None values. Update test cases to reflect changes in tag values.
Enhance Redis activity by formatting set and get operations for improved readability and consistency. Update tests to include metadata in assertions for better verification of Redis interactions.
Refactor notification methods in Gates, MongoDB, and Redis classes to use asynchronous send_notification_async. Update related tests to ensure proper async handling and verification of notifications.
Refactor initialization of Activities, Gates, and MongoDB classes for improved readability by using multi-line argument formatting. Update related tests to match the new initialization style.
Update environment configuration and refactor activities to include metrics controller. Remove Kafka settings and adjust Redis and MongoDB initialization. Update tests to reflect changes in initialization and metrics tracking.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1325 for specific external operation metrics. Refactor shutdown methods in Activities class to ensure proper closure of MongoDB, Redis, and Gates connections. Adjust tests to reflect these changes and enhance repository pattern implementation for MongoDB and Redis activities.
SIENTIAPDE-1314: Update workflow name from 'sub_core_scouter' to 'subworkflow.core_scouter' in scouter.py and core_scouter.py, and adjust test references accordingly in test_scouter.py for consistency.
Implement tag value registration in Gates activity and update related tests. Enhanced the write_metrics method to process and register tag values, improving metrics tracking. Adjusted tests to validate new functionality.
Update Redis and CoreScouter to incorporate 'fill_missing_tags' parameter in data processing. Adjusted related tests to ensure proper handling of missing tags, enhancing overall functionality and consistency across workflows.
Update .gitignore and refactor metrics.py, activities.py, and gates.py for improved clarity and consistency. Added coverage.xml and cache directories to .gitignore. Standardized string formatting and parameter handling in metrics and activities classes, enhancing code readability. Removed the deprecated faker.py file and adjusted related tests accordingly.
Update test_activities.py to change assertions for mock_postgres_close and mock_mongodb_close from assert_called_once to assert_called, allowing for more flexible verification of calls during shutdown tests.
Enhance timestamp handling in Redis and CoreScouter workflows by updating to DATETIME_FORMAT_MS_WITH_TZ. Modify tests to reflect new timestamp format, ensuring consistency across data structures and improving overall datetime management.
Update sientia-dataops-library version to 1.4.3 and refactor timestamp handling in MongoDB and Redis activities to use DATETIME_FORMAT constants for improved consistency. Enhance unit tests to reflect these changes.
SIENTIAPDE-1169 Update Redis and core scouter tests to include model_tags in data structures, enhancing data organization and consistency across test cases.
Update sonar-project.properties to exclude worker.py from coverage, enhance error handling in worker.py, and add new tests for error scenarios in test_redis.py and test_mongo.py.
Refactor test cases in test_gates.py to update filter configurations, extracting policy into a dictionary format for improved clarity and consistency in data quality gate tests.
Implement store_data_package method in Redis activity for debug data storage. Update CoreScouter workflow to conditionally call store_data_package based on debug flag. Enhance tests for store_data_package functionality and error handling.
Enhance Gates and Redis activities by adding metadata parameter to apply_aggregation and notification methods. Refactor notification handling to use send_notification for improved consistency. Update tests to reflect changes in notification method calls and ensure proper functionality with new metadata integration.
Refactor Activities class to remove Kafka and Druid dependencies, simplifying initialization. Update values.yaml to set replica count to 1 for reduced resource usage. Adjust Redis activity to set TTL to None for better data retention. Remove unused Kafka and Druid activity files and their associated tests, streamlining the codebase.
Update sientia-dataops-library version to 1.2.0 in requirements.txt; refactor logging in activities to use a unified Logger instance and include metadata in log messages across various activities.
Implement workflows for fake data generation, scouter processing, and core scouter operations
- Added `FakeData` workflow to generate random data and send it to a Kafka topic.
- Implemented `Scouter` workflow to load data from Kafka and trigger the core scouter workflow.
- Created `CoreScouter` workflow to process data through quality gates, aggregation, and export to PostgreSQL.
- Developed comprehensive unit tests for activities and workflows, ensuring proper functionality and error handling.
- Enhanced Redis and Postgres activities with robust testing for data handling and error notifications.
- Introduced quality filters for data validation and implemented tests to verify their functionality.
