Refactor imports and enhance timestamp handling in API and test files
- Removed redundant import statements in api.py and prepare_worker.py for cleaner code.
- Updated timestamp format in test cases to include timezone information for consistency.
- Adjusted test configurations to align with new API structure for improved accuracy in data handling.
Enhance PI Web API integration and update test configurations
- Adjusted execution counts in tests.ipynb for better notebook state management.
- Introduced a new pi_web_api_query structure in tests.ipynb for improved API configuration.
- Updated API class to format timestamps using DATETIME_FORMAT_WITH_TZ.
- Refactored PIWebAPIClient to ensure correct web ID retrieval and emit metrics accurately.
- Streamlined workflow parameters in pi_web_api_scouter.py for clarity and flexibility.
SIENTIAPDE-1445 Add PI Web API configuration to .env.example and enhance worker setup
- Introduced new environment variables for PI Web API configuration in .env.example.
- Updated prepare_worker.py to utilize a camelCase to kebab-case conversion function for queue naming.
- Enhanced logging in prepare_worker to provide better insights during worker preparation.
- Adjusted execution counts in tests.ipynb for improved notebook state management.
Refactor workflow parameters in tests.ipynb and prepare_worker.py to enhance clarity and flexibility. Updated workflow_id and task_queue to accept dynamic values, and standardized timeout parameters for improved configuration.
Enhance Activities and API Integration
- Updated Activities class to include API operations for external data ingestion.
- Added API configuration builder to connectors_config.py for environment variable management.
- Integrated API configuration into worker setup.
- Expanded unit tests to cover new API functionality and configuration handling.
- Updated requirements.txt to include pycurl and prometheus-client for enhanced metrics support.
Refactor prepare_worker.py to improve type annotations and code clarity. Update import statements and enhance formatting for better readability. Adjust comments in worker.py for consistency.
SIENTIAPDE-1441: Refactor worker configuration in worker.py to utilize a new prepare_worker function for cleaner setup. Update environment variable names in values.yaml for consistency and clarity, adjusting memory requests and enhancing health check commands for improved application monitoring.
Update values.yaml to modify liveness and readiness probe configurations, enhancing application health checks with improved metrics validation and adjusted timing parameters for better responsiveness.
Update README.md and values.yaml to enhance system description and worker configuration. Added ML model orchestration to the system overview, introduced new debug mode features, and updated environment variable values to be strings for consistency.
SIENTIAPDE-1442: Enhance Temporal worker configuration by adding environment variables for concurrency limits and poller behavior settings in values.yaml and worker.py for optimized performance.
Remove encrypt.py and test.ipynb files; update values.yaml to set image tag to 1.0.0 and change GITHUB_BRANCH to release/SIENTIAPDE-1441 for version consistency.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1273, increment Helm chart version to 0.6.0, and modify metrics registration in Gates class to skip None values. Update test cases to reflect changes in tag values.
Enhance Redis activity by formatting set and get operations for improved readability and consistency. Update tests to include metadata in assertions for better verification of Redis interactions.
Update requirements to use the remote sientia-dataops-library repository and increment image tag in values.yaml from 0.4.9 to 0.5.0 for version consistency.
Refactor notification methods in Gates, MongoDB, and Redis classes to use asynchronous send_notification_async. Update related tests to ensure proper async handling and verification of notifications.
Refactor initialization of Activities, Gates, and MongoDB classes for improved readability by using multi-line argument formatting. Update related tests to match the new initialization style.
Update environment configuration and refactor activities to include metrics controller. Remove Kafka settings and adjust Redis and MongoDB initialization. Update tests to reflect changes in initialization and metrics tracking.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1325 for specific external operation metrics. Refactor shutdown methods in Activities class to ensure proper closure of MongoDB, Redis, and Gates connections. Adjust tests to reflect these changes and enhance repository pattern implementation for MongoDB and Redis activities.
Refactor MongoDB, Redis, and Gates classes to extend SientiaMonitoring instead of BaseActivity. Update requirements to point to local dataops library path. Implement repository pattern for MongoDB and Redis operations, enhancing code organization and maintainability.
SIENTIAPDE-1314: Update workflow name from 'sub_core_scouter' to 'subworkflow.core_scouter' in scouter.py and core_scouter.py, and adjust test references accordingly in test_scouter.py for consistency.
SIENTIAPDE-1314: Update type annotations in encrypt.py for improved clarity and type safety. Added type hint for file_tree variable and included type ignore for yaml import.
Implement tag value registration in Gates activity and update related tests. Enhanced the write_metrics method to process and register tag values, improving metrics tracking. Adjusted tests to validate new functionality.
Update requirements and simplify CI workflow by integrating quality gate from template. Bump sientia-dataops-library version to 1.4.7 and streamline quality-gate.yml to use a shared workflow configuration.
Update Redis and CoreScouter to incorporate 'fill_missing_tags' parameter in data processing. Adjusted related tests to ensure proper handling of missing tags, enhancing overall functionality and consistency across workflows.
SIENTIAPDE-1318: Add 'fill_missing_tags' parameter to CoreScouter for enhanced tag handling in data processing. This update allows for the specification of missing tags during workflow execution.
SIENTIAPDE-1318: Add support for filling missing tags in Redis data processing. Enhanced the Redis class to include a new parameter for handling missing tags, ensuring that absent tags are filled with None in the data package.
Enhance validation script and CI workflow for code quality checks
- Added support for a --fix option in validate.sh to apply Ruff auto-fixes for code formatting and linting.
- Updated quality-gate.yml to include separate steps for installing development and runtime dependencies.
- Introduced dedicated steps for code formatting checks, linting, type checking, and security analysis in the CI workflow.
