Enhance validation script and CI workflow for code quality checks
- Added support for a --fix option in validate.sh to apply Ruff auto-fixes for code formatting and linting.
- Updated quality-gate.yml to include separate steps for installing development and runtime dependencies.
- Introduced dedicated steps for code formatting checks, linting, type checking, and security analysis in the CI workflow.
Update .gitignore and refactor metrics.py, activities.py, and gates.py for improved clarity and consistency. Added coverage.xml and cache directories to .gitignore. Standardized string formatting and parameter handling in metrics and activities classes, enhancing code readability. Removed the deprecated faker.py file and adjusted related tests accordingly.
SIENTIAPDE-1084: Refactor apply_aggregation method in Gates class for improved performance and clarity. Changed parameter name from 'group' to 'values', optimized aggregation logic, and enhanced error handling for invalid functions. Streamlined data processing by reducing unnecessary operations and added batch debug logging.
Update test_activities.py to change assertions for mock_postgres_close and mock_mongodb_close from assert_called_once to assert_called, allowing for more flexible verification of calls during shutdown tests.
Update README.md to modify the Scouter Workflow flowchart, replacing the icon for the core_scouter step with a refresh symbol for better visual representation of data processing continuity.
Update README.md to correct the flowchart syntax for the Scouter Workflow Architecture, changing from 'flowchart-elk' to 'flowchart' to ensure proper rendering and maintain clarity in visualizing data processing steps.
Update README.md to modify the Scouter Workflow Architecture flowchart, changing the layout to 'flowchart-elk' for improved clarity in visualizing data processing steps and interactions.
Refactor README.md to update the Scouter Workflow Architecture flowchart, transitioning to a left-to-right layout for improved readability and clarity in depicting data processing steps and service interactions.
Update README.md to revise the Scouter Workflow Architecture flowchart, replacing outdated steps with a new data processing sequence that includes data quality checks, aggregation, and metrics writing, enhancing clarity on system operations and service interactions.
Update README.md to enhance the architecture diagrams for both Scouter and FakeData workflows, improving clarity and structure by adopting a consistent flowchart format and detailing service interactions.
Update README.md to modify the Scouter Workflow Architecture flowchart, enhancing clarity by restructuring the diagram layout and correcting the connections to Redis and MongoDB components.
Refactor README.md to streamline the Scouter Workflow Architecture section by consolidating the flowchart into a single line format for improved readability and clarity.
Update README.md to include a new section on Scouter Workflow Architecture, detailing the incremental data processing steps and key components, while removing outdated system architecture diagrams for improved clarity and relevance.
Enhance README.md by adding detailed system architecture and data processing flow diagrams using Mermaid syntax, improving clarity on the Scouter system's components and interactions.
Update README.md to change the section title from "Temporal input parameters" to "MongoDB pipeline configuration" for improved accuracy in documentation.
Update README.md to reflect new workflow configuration parameters, including temporal input settings, tag definitions, and filter policies for enhanced clarity on system capabilities.
Remove deprecated files and configurations, including .env, Dockerfile, docker-compose.yml, and client-schedule.py. Update README.md to reflect new architecture and features, enhancing clarity on system capabilities and workflows. Adjust values.yaml for image tag and replica count, and improve code documentation across various modules for better maintainability.
Refactor timestamp handling in CoreScouter workflow by updating to DATETIME_FORMAT_WITH_TZ, ensuring consistency with recent changes in datetime management.
Enhance timestamp handling in Redis and CoreScouter workflows by updating to DATETIME_FORMAT_MS_WITH_TZ. Modify tests to reflect new timestamp format, ensuring consistency across data structures and improving overall datetime management.
Refactor Redis activity to utilize the new 'now' function for timestamp generation, ensuring consistency with updated datetime handling. This change replaces the direct use of 'datetime.now()' with 'now()' from the temporal constants.
Update sientia-dataops-library version to 1.4.3 and refactor timestamp handling in MongoDB and Redis activities to use DATETIME_FORMAT constants for improved consistency. Enhance unit tests to reflect these changes.
Update requirements and core scouter workflow: comment out old dataops library dependency, change default namespace in worker.py, and enhance timestamp handling in core_scouter.py with DATETIME_FORMAT.
Refactor logging in worker.py to use custom_info and custom_error methods, enhancing log metadata with pod and workflow details for improved observability.
Enhance logging in Gates, MongoDB, and Redis activities by replacing debug statements with info level logs, improving observability of data processing steps. Update worker configuration to adjust concurrency settings and enable autoscaling for task polling.
Update dependencies and improve observability by changing logger imports. Bump sientia-dataops-library version to 1.4.1 and update image tag to 0.4.2 in values.yaml.
SIENTIAPDE-1169 Update Redis and core scouter tests to include model_tags in data structures, enhancing data organization and consistency across test cases.
Enable SDK metrics service in values.yaml and update worker.py to initialize SDK metrics server with Prometheus configuration, enhancing observability of the application.
Update image tag to 0.4.0 in values.yaml, modify GITHUB_BRANCH for SIENTIAPDE-1169, and enhance Redis activity to handle model tags for improved data retention.
