Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations
- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
SIENTIAPDE-1325: Refactor Ingestor and Manager Classes for Enhanced Asynchronous Operations
- Introduced asynchronous methods across Ingestor, IngestorManager, DataManager, and OpcManager to improve performance and responsiveness.
- Integrated MetricsController into various classes for better observability and monitoring.
- Updated Redis and MongoDB interactions to support asynchronous operations, enhancing data handling efficiency.
- Removed deprecated Redis metrics and streamlined resource management logic.
- Adjusted unit tests to accommodate the new asynchronous behavior and ensure proper mocking of async methods.
Refactor IngestorManager and OpcManager for improved server management
- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
Updated the Ingestor and IngestorManager classes to use async methods for checking OPC server integrity and shutting down OPC managers. Adjusted related tests to ensure proper mocking of asynchronous behavior.
Update image tag in values.yaml and refactor OpcManager error handling
- Updated the image tag in values.yaml from "0.4.5" to "0.4.9" for the latest version.
- Refactored error handling in OpcManager to improve logging and maintain code clarity.
- Adjusted unit tests for better readability and consistency in assertions.
Refactor OpcManager connection error handling and enhance unit tests
- Simplified error handling during connection attempts in OpcManager by removing redundant disconnection logic.
- Updated unit tests to include subscription_period_ms in server configuration and adjusted connection assertions to include a timeout parameter.
- Added new tests for disconnection fallback functionality to ensure robust error handling during disconnect attempts.
Enhance OpcManager to accept subscription period from configuration
- Updated OpcManager to include subscription_period_ms as a parameter for better configurability.
- Refactored create_subscription method to utilize the new subscription period parameter.
- Adjusted unit tests to reflect changes in subscription handling and ensure correct functionality.
Refactor project structure and update configurations
- Deleted the empty `__init__.py` file to clean up the project structure.
- Renamed the project from "laborious" to "ingestor" in `pyproject.toml`, updating the description accordingly.
- Improved type hints and conditional checks in the `Ingestor`, `DataManager`, `IngestorManager`, and `OpcManager` classes for better code clarity and type safety.
- Enhanced error handling and assertions in various methods to ensure robustness.
- Updated unit tests to reflect changes in class names and error handling improvements.
- Updated the `sientia-dataops-library` dependency version from 1.4.3 to 1.4.6 in `requirements.txt`.
- Modified the GitHub Actions workflow to install development and runtime dependencies separately, improving clarity and organization.
- Added code formatting and linting checks using Ruff, along with type checking using mypy, to ensure code quality.
- Updated `.gitignore` to include additional cache directories and log files.
- Refactored code in various files for consistency in string formatting and improved logging messages.
Enhance unit tests for async shutdown and improve assertions
- Updated `test_app.py` to verify multiple calls to `asyncio_sleep` and ensure proper handling of sleep intervals.
- Added new tests in `test_ingestor.py` and `test_opc_manager.py` to validate shutdown behavior and error handling for async operations.
- Improved assertions in existing tests to enhance reliability and clarity of test outcomes.
Refactor tests for async compatibility and enhance IngestorManager functionality
- Updated test cases in `test_app.py`, `test_ingestor.py`, and `test_ingestor_manager.py` to use async/await syntax for improved concurrency.
- Refactored methods in IngestorManager and related classes to support asynchronous operations, ensuring non-blocking behavior during execution.
- Enhanced mock setups in tests to accommodate async methods, improving test reliability and performance.
Update requirements and refactor datetime handling in OpcManager for timezone support
- Commented out the specific version of sientia-dataops-library in requirements.txt and added a local path.
- Updated GITHUB_BRANCH in values.yaml for consistency with current development.
- Refactored source_timestamp handling in OpcManager to include timezone information.
- Adjusted timestamp formatting in test cases to reflect the new datetime format with timezone.
Refactor test assertions in test_opc_manager.py for improved readability
- Updated the assertion syntax for notification calls in the test_check_opc_listenning_error_notification_and_retry function to use dictionary literals for clarity.
Update requirements and refactor manager classes to remove pod_id
- Updated the sientia-dataops-library dependency version to 1.3.8 in requirements.txt.
- Refactored IngestorManager, OpcManager, and ResourceManager classes to remove pod_id from their initialization and internal handling, enhancing code clarity and consistency.
- Adjusted unit tests to reflect the removal of pod_id, ensuring they remain functional and accurate.
Refactor unit tests to improve readability and maintainability
- Updated test assertions and mock setups in test_app.py for clarity.
- Added 'pod_id' to the metadata in test_ingestor.py for consistency.
- Introduced a new test for TAG_WRITTEN_COUNT in test_metrics.py to verify its definition.
- Enhanced test data structure in test_data_manager.py to include additional fields.
SIENTIAPDE-1163
Refactor Ingestor and DataManager classes to enhance metadata handling
- Updated Ingestor class to include additional metadata fields: 'workflow_name' and 'schema_name'.
- Modified DataManager to accept metadata during initialization.
- Adjusted unit tests to validate the new metadata structure and ensure proper functionality across various managers.
- Removed unused logger initialization in tests for cleaner code.
Refactor test assertions for improved readability and consistency
- Updated assertions in test cases for DataManager and OpcManager to enhance readability by formatting long lines.
- Ensured that the expected behavior of metrics logging and message publishing is clearly defined in the tests.
- Updated DataManager to simplify MongoDB connection logic by removing retry loop and adding logging for connection attempts.
- Enhanced IngestorManager to accept MongoDB connection parameters and export_to_kafka flag as part of its initialization.
- Updated requirements.txt to use the latest version of the sientia-dataops-library.
Update ingestor and data manager for improved shutdown handling and logging
- Incremented project version in quality gate configuration.
- Commented out ingestor service in docker-compose for clarity.
- Enhanced main loop in app.py to handle exit signals and exceptions.
- Added shutdown methods in Ingestor and DataManager classes for graceful resource cleanup.
- Updated unit tests to validate shutdown behavior and exception handling.
- Introduced coverage configuration to omit specific files.
Enhance IngestorManager and related tests with notification handling integration
- Added notification_handler parameter to IngestorManager and OpcManager initialization for improved event handling.
- Updated unit tests to include notification_handler in the Ingestor and IngestorManager instantiation, ensuring proper coverage of new functionality.
- Refactored test cases to validate the integration of notification handling across components.
Enhance ingestor and manager classes with notification handling
- Integrated NotificationHandler into Ingestor, DataManager, IngestorManager, and OpcManager for improved error reporting and monitoring.
- Updated methods to send notifications on critical events such as Kafka publishing errors, OPC connection issues, and cycle count warnings.
- Refactored related tests to ensure coverage of new notification functionalities and validate integration with existing components.
- Improved logging and error handling across the system to enhance traceability and operational insights.
Refactor ingestor lease management and update Docker configurations
- Updated CMD in Dockerfile to use the correct application entry point.
- Added docker compose command to remove volumes in README.
- Removed unnecessary restart policy from ingestor service in docker-compose.yaml.
- Refactored manage_leases method in Ingestor class to improve clarity and parameter naming.
- Added get_number_of_leases method in IngestorManager to retrieve active leases.
- Implemented get_all_leases method in ResourceManager to fetch active leases from Redis.
- Enhanced unit tests for lease management in test_ingestor_manager and test_resource_manager.
Refactor OpcManager initialization and unsubscribe method; enhance logging
- Removed the `initialize_from_config` method from `OpcManager` class and adjusted the constructor to handle configuration directly.
- Improved the `unsubscribe` method to include detailed logging for non-existent subscriptions.
- Updated tests in `test_opc_manager.py` to reflect changes in the `OpcManager` class.
- Commented out Docker-related fixtures in `conftest.py` for potential future use.
- Enhanced test coverage in `test_single_node.py` and `test_data_manager.py` with additional scenarios and assertions.
- Introduced new methods in `Ingestor` and `IngestorManager` classes to manage server subscriptions and leases more effectively.
- Added new tests for `Ingestor` class to validate initialization and slot management logic.
- Implemented logging improvements across various classes to ensure better traceability of actions and errors.
Add temporary functiontal tests, that must be removed soon.
Refactor tests: Remove outdated resource manager tests and add functional tests with Docker and Kafka integration
- Deleted existing unit tests for ResourceManager.
- Introduced new functional tests for single node operations with Redis and Kafka.
- Added Docker Compose setup for test environment.
- Implemented fixtures for Redis and Kafka consumers.
- Created comprehensive tests for data publishing and slot management.
- Added unit tests for DataManager and IngestorManager with mocked dependencies.
- Included tests for OpcManager covering connection, subscription, and data change notifications.
