Update requirements and refactor manager classes to remove pod_id
- Updated the sientia-dataops-library dependency version to 1.3.8 in requirements.txt.
- Refactored IngestorManager, OpcManager, and ResourceManager classes to remove pod_id from their initialization and internal handling, enhancing code clarity and consistency.
- Adjusted unit tests to reflect the removal of pod_id, ensuring they remain functional and accurate.
Refactor unit tests to improve readability and maintainability
- Updated test assertions and mock setups in test_app.py for clarity.
- Added 'pod_id' to the metadata in test_ingestor.py for consistency.
- Introduced a new test for TAG_WRITTEN_COUNT in test_metrics.py to verify its definition.
- Enhanced test data structure in test_data_manager.py to include additional fields.
SIENTIAPDE-1163
Refactor Ingestor and DataManager classes to enhance metadata handling
- Updated Ingestor class to include additional metadata fields: 'workflow_name' and 'schema_name'.
- Modified DataManager to accept metadata during initialization.
- Adjusted unit tests to validate the new metadata structure and ensure proper functionality across various managers.
- Removed unused logger initialization in tests for cleaner code.
Refactor test assertions for improved readability and consistency
- Updated assertions in test cases for DataManager and OpcManager to enhance readability by formatting long lines.
- Ensured that the expected behavior of metrics logging and message publishing is clearly defined in the tests.
- Updated DataManager to simplify MongoDB connection logic by removing retry loop and adding logging for connection attempts.
- Enhanced IngestorManager to accept MongoDB connection parameters and export_to_kafka flag as part of its initialization.
- Updated requirements.txt to use the latest version of the sientia-dataops-library.
Update ingestor and data manager for improved shutdown handling and logging
- Incremented project version in quality gate configuration.
- Commented out ingestor service in docker-compose for clarity.
- Enhanced main loop in app.py to handle exit signals and exceptions.
- Added shutdown methods in Ingestor and DataManager classes for graceful resource cleanup.
- Updated unit tests to validate shutdown behavior and exception handling.
- Introduced coverage configuration to omit specific files.
Enhance IngestorManager and related tests with notification handling integration
- Added notification_handler parameter to IngestorManager and OpcManager initialization for improved event handling.
- Updated unit tests to include notification_handler in the Ingestor and IngestorManager instantiation, ensuring proper coverage of new functionality.
- Refactored test cases to validate the integration of notification handling across components.
Enhance ingestor and manager classes with notification handling
- Integrated NotificationHandler into Ingestor, DataManager, IngestorManager, and OpcManager for improved error reporting and monitoring.
- Updated methods to send notifications on critical events such as Kafka publishing errors, OPC connection issues, and cycle count warnings.
- Refactored related tests to ensure coverage of new notification functionalities and validate integration with existing components.
- Improved logging and error handling across the system to enhance traceability and operational insights.
Refactor ingestor lease management and update Docker configurations
- Updated CMD in Dockerfile to use the correct application entry point.
- Added docker compose command to remove volumes in README.
- Removed unnecessary restart policy from ingestor service in docker-compose.yaml.
- Refactored manage_leases method in Ingestor class to improve clarity and parameter naming.
- Added get_number_of_leases method in IngestorManager to retrieve active leases.
- Implemented get_all_leases method in ResourceManager to fetch active leases from Redis.
- Enhanced unit tests for lease management in test_ingestor_manager and test_resource_manager.
Refactor OpcManager initialization and unsubscribe method; enhance logging
- Removed the `initialize_from_config` method from `OpcManager` class and adjusted the constructor to handle configuration directly.
- Improved the `unsubscribe` method to include detailed logging for non-existent subscriptions.
- Updated tests in `test_opc_manager.py` to reflect changes in the `OpcManager` class.
- Commented out Docker-related fixtures in `conftest.py` for potential future use.
- Enhanced test coverage in `test_single_node.py` and `test_data_manager.py` with additional scenarios and assertions.
- Introduced new methods in `Ingestor` and `IngestorManager` classes to manage server subscriptions and leases more effectively.
- Added new tests for `Ingestor` class to validate initialization and slot management logic.
- Implemented logging improvements across various classes to ensure better traceability of actions and errors.
Add temporary functiontal tests, that must be removed soon.
Refactor tests: Remove outdated resource manager tests and add functional tests with Docker and Kafka integration
- Deleted existing unit tests for ResourceManager.
- Introduced new functional tests for single node operations with Redis and Kafka.
- Added Docker Compose setup for test environment.
- Implemented fixtures for Redis and Kafka consumers.
- Created comprehensive tests for data publishing and slot management.
- Added unit tests for DataManager and IngestorManager with mocked dependencies.
- Included tests for OpcManager covering connection, subscription, and data change notifications.
Enhance README and Docker setup; improve IngestorManager and DataManager functionality, add error handling, and implement unit tests for OPC initialization and subscription management.
Implement initial structure for data ingestion with Kafka and OPC UA integration
- Added DataManager for Kafka message handling
- Introduced IngestorManager to manage data ingestion processes
- Created OpcManager for OPC UA client interactions
- Developed ResourceManager for Redis-based resource management
- Established testing framework with unit tests for DataManager and OpcManager
- Configured project settings for Python testing with pytest
