Enhance release workflow to trigger only on merged pull requests and streamline data insertion in DataManager
- Updated the release workflow to execute only when a pull request is merged.
- Simplified the document insertion logic in DataManager by consolidating the dictionary construction.
- Refactored OPC Manager tests to utilize asynchronous mocks for improved accuracy in testing.
Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations
- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
Refactor IngestorManager and OpcManager for improved server management
- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
Update image tag in values.yaml and refactor OpcManager error handling
- Updated the image tag in values.yaml from "0.4.5" to "0.4.9" for the latest version.
- Refactored error handling in OpcManager to improve logging and maintain code clarity.
- Adjusted unit tests for better readability and consistency in assertions.
Refactor OpcManager connection error handling and enhance unit tests
- Simplified error handling during connection attempts in OpcManager by removing redundant disconnection logic.
- Updated unit tests to include subscription_period_ms in server configuration and adjusted connection assertions to include a timeout parameter.
- Added new tests for disconnection fallback functionality to ensure robust error handling during disconnect attempts.
Enhance OpcManager to accept subscription period from configuration
- Updated OpcManager to include subscription_period_ms as a parameter for better configurability.
- Refactored create_subscription method to utilize the new subscription period parameter.
- Adjusted unit tests to reflect changes in subscription handling and ensure correct functionality.
Refactor project structure and update configurations
- Deleted the empty `__init__.py` file to clean up the project structure.
- Renamed the project from "laborious" to "ingestor" in `pyproject.toml`, updating the description accordingly.
- Improved type hints and conditional checks in the `Ingestor`, `DataManager`, `IngestorManager`, and `OpcManager` classes for better code clarity and type safety.
- Enhanced error handling and assertions in various methods to ensure robustness.
- Updated unit tests to reflect changes in class names and error handling improvements.
- Updated the `sientia-dataops-library` dependency version from 1.4.3 to 1.4.6 in `requirements.txt`.
- Modified the GitHub Actions workflow to install development and runtime dependencies separately, improving clarity and organization.
- Added code formatting and linting checks using Ruff, along with type checking using mypy, to ensure code quality.
- Updated `.gitignore` to include additional cache directories and log files.
- Refactored code in various files for consistency in string formatting and improved logging messages.
Enhance unit tests for async shutdown and improve assertions
- Updated `test_app.py` to verify multiple calls to `asyncio_sleep` and ensure proper handling of sleep intervals.
- Added new tests in `test_ingestor.py` and `test_opc_manager.py` to validate shutdown behavior and error handling for async operations.
- Improved assertions in existing tests to enhance reliability and clarity of test outcomes.
Refactor tests for async compatibility and enhance IngestorManager functionality
- Updated test cases in `test_app.py`, `test_ingestor.py`, and `test_ingestor_manager.py` to use async/await syntax for improved concurrency.
- Refactored methods in IngestorManager and related classes to support asynchronous operations, ensuring non-blocking behavior during execution.
- Enhanced mock setups in tests to accommodate async methods, improving test reliability and performance.
Update requirements and refactor datetime handling in OpcManager for timezone support
- Commented out the specific version of sientia-dataops-library in requirements.txt and added a local path.
- Updated GITHUB_BRANCH in values.yaml for consistency with current development.
- Refactored source_timestamp handling in OpcManager to include timezone information.
- Adjusted timestamp formatting in test cases to reflect the new datetime format with timezone.
Refactor test assertions in test_opc_manager.py for improved readability
- Updated the assertion syntax for notification calls in the test_check_opc_listenning_error_notification_and_retry function to use dictionary literals for clarity.
Update requirements and refactor manager classes to remove pod_id
- Updated the sientia-dataops-library dependency version to 1.3.8 in requirements.txt.
- Refactored IngestorManager, OpcManager, and ResourceManager classes to remove pod_id from their initialization and internal handling, enhancing code clarity and consistency.
- Adjusted unit tests to reflect the removal of pod_id, ensuring they remain functional and accurate.
SIENTIAPDE-1163
Refactor Ingestor and DataManager classes to enhance metadata handling
- Updated Ingestor class to include additional metadata fields: 'workflow_name' and 'schema_name'.
- Modified DataManager to accept metadata during initialization.
- Adjusted unit tests to validate the new metadata structure and ensure proper functionality across various managers.
- Removed unused logger initialization in tests for cleaner code.
Refactor test assertions for improved readability and consistency
- Updated assertions in test cases for DataManager and OpcManager to enhance readability by formatting long lines.
- Ensured that the expected behavior of metrics logging and message publishing is clearly defined in the tests.
Update ingestor and data manager for improved shutdown handling and logging
- Incremented project version in quality gate configuration.
- Commented out ingestor service in docker-compose for clarity.
- Enhanced main loop in app.py to handle exit signals and exceptions.
- Added shutdown methods in Ingestor and DataManager classes for graceful resource cleanup.
- Updated unit tests to validate shutdown behavior and exception handling.
- Introduced coverage configuration to omit specific files.
Enhance ingestor and manager classes with notification handling
- Integrated NotificationHandler into Ingestor, DataManager, IngestorManager, and OpcManager for improved error reporting and monitoring.
- Updated methods to send notifications on critical events such as Kafka publishing errors, OPC connection issues, and cycle count warnings.
- Refactored related tests to ensure coverage of new notification functionalities and validate integration with existing components.
- Improved logging and error handling across the system to enhance traceability and operational insights.
Refactor OpcManager initialization and unsubscribe method; enhance logging
- Removed the `initialize_from_config` method from `OpcManager` class and adjusted the constructor to handle configuration directly.
- Improved the `unsubscribe` method to include detailed logging for non-existent subscriptions.
- Updated tests in `test_opc_manager.py` to reflect changes in the `OpcManager` class.
- Commented out Docker-related fixtures in `conftest.py` for potential future use.
- Enhanced test coverage in `test_single_node.py` and `test_data_manager.py` with additional scenarios and assertions.
- Introduced new methods in `Ingestor` and `IngestorManager` classes to manage server subscriptions and leases more effectively.
- Added new tests for `Ingestor` class to validate initialization and slot management logic.
- Implemented logging improvements across various classes to ensure better traceability of actions and errors.
Add temporary functiontal tests, that must be removed soon.
Refactor tests: Remove outdated resource manager tests and add functional tests with Docker and Kafka integration
- Deleted existing unit tests for ResourceManager.
- Introduced new functional tests for single node operations with Redis and Kafka.
- Added Docker Compose setup for test environment.
- Implemented fixtures for Redis and Kafka consumers.
- Created comprehensive tests for data publishing and slot management.
- Added unit tests for DataManager and IngestorManager with mocked dependencies.
- Included tests for OpcManager covering connection, subscription, and data change notifications.
