Enhance OpcManager to accept subscription period from configuration
- Updated OpcManager to include subscription_period_ms as a parameter for better configurability.
- Refactored create_subscription method to utilize the new subscription period parameter.
- Adjusted unit tests to reflect changes in subscription handling and ensure correct functionality.
Refactor project structure and update configurations
- Deleted the empty `__init__.py` file to clean up the project structure.
- Renamed the project from "laborious" to "ingestor" in `pyproject.toml`, updating the description accordingly.
- Improved type hints and conditional checks in the `Ingestor`, `DataManager`, `IngestorManager`, and `OpcManager` classes for better code clarity and type safety.
- Enhanced error handling and assertions in various methods to ensure robustness.
- Updated unit tests to reflect changes in class names and error handling improvements.
- Updated the `sientia-dataops-library` dependency version from 1.4.3 to 1.4.6 in `requirements.txt`.
- Modified the GitHub Actions workflow to install development and runtime dependencies separately, improving clarity and organization.
- Added code formatting and linting checks using Ruff, along with type checking using mypy, to ensure code quality.
- Updated `.gitignore` to include additional cache directories and log files.
- Refactored code in various files for consistency in string formatting and improved logging messages.
Enhance documentation and improve shutdown methods for DataManager and IngestorManager
- Updated README.md to clarify persistent storage capabilities.
- Enhanced docstrings in DataManager and IngestorManager for shutdown processes, detailing resource cleanup and error handling.
- Improved comments in various methods to provide clearer explanations of functionality and side effects.
Remove deprecated files and enhance documentation
- Deleted `coverage.sh`, `docker-compose.yaml`, `Dockerfile`, and simulator-related files to streamline the project structure.
- Updated `README.md` to provide a comprehensive overview of the OPC Ingestor, including features, architecture, installation, usage, and troubleshooting.
- Enhanced docstrings across various classes and methods in the `ingestor` module for better clarity and maintainability.
- Improved Prometheus metrics documentation in `metrics.py` to ensure proper monitoring and observability of the system.
Update values.yaml and refactor shutdown methods for improved configuration and async handling
- Changed replica count from 2 to 1 in values.yaml for deployment optimization.
- Updated image tag from 0.4.3 to 0.4.5 in values.yaml for version consistency.
- Refactored shutdown method in IngestorManager to use await server.shutdown() for better async operation.
- Removed __del__ method in OpcManager to prevent blocking during cleanup.
Refactor Ingestor and OPC Manager for Asynchronous Operations
- Updated main function to be asynchronous and integrated asyncio for better concurrency.
- Refactored Ingestor methods to support async operations, including prepare_ingestor, loop, and shutdown.
- Enhanced IngestorManager and OpcManager with async methods for improved performance and responsiveness.
- Replaced blocking calls with await statements to ensure non-blocking behavior during operations.
- Added a new run_async_main function to handle the async event loop setup.
Update requirements and refactor datetime handling in OpcManager for timezone support
- Commented out the specific version of sientia-dataops-library in requirements.txt and added a local path.
- Updated GITHUB_BRANCH in values.yaml for consistency with current development.
- Refactored source_timestamp handling in OpcManager to include timezone information.
- Adjusted timestamp formatting in test cases to reflect the new datetime format with timezone.
- Updated sientia-dataops-library version to 1.4.1 in requirements.txt.
- Changed image tag from 0.3.2 to 0.4.2 in values.yaml.
- Refactored logger imports from sientia_do.temporal.utils.logger to sientia_do.observability.logger in multiple manager files for improved observability.
Update requirements and refactor manager classes to remove pod_id
- Updated the sientia-dataops-library dependency version to 1.3.8 in requirements.txt.
- Refactored IngestorManager, OpcManager, and ResourceManager classes to remove pod_id from their initialization and internal handling, enhancing code clarity and consistency.
- Adjusted unit tests to reflect the removal of pod_id, ensuring they remain functional and accurate.
Update BaseActivity initialization in manager classes to enable error counter metric
- Modified the initialization of DataManager, IngestorManager, OpcManager, and ResourceManager to include 'set_error_counter=True' for enhanced error tracking.
Update BaseActivity initialization in manager classes to disable error counter metric
- Modified the initialization of DataManager, IngestorManager, OpcManager, and ResourceManager to include 'set_error_counter_metric=False' for improved error handling.
Refactor Ingestor and Manager classes to use CoreNotificationHandler and improve metadata handling
- Updated Ingestor class to initialize logger using get_logger and replaced notification handler initialization with metadata dictionary.
- Refactored DataManager, IngestorManager, OpcManager, and ResourceManager classes to inherit from BaseActivity, allowing for consistent logger and notification handler usage.
- Enhanced notification handling by integrating metadata into notification methods across various managers.
- Added .vscode/ to .gitignore to exclude VSCode configuration files.
Refactor OpcManager to improve node subscription handling
- Simplified the address node assignment in the OpcManager class by directly using a local variable instead of an instance variable.
- Enhanced debug logging to provide clearer insights into the nodes being subscribed to.
Refactor OPC Manager metrics logging for improved readability
- Reformatted metrics logging statements in the OpcManager class for better readability and consistency.
- Enhanced debug logging to provide clearer insights into subscription management and connection status.
Update ingestor and data manager for improved shutdown handling and logging
- Incremented project version in quality gate configuration.
- Commented out ingestor service in docker-compose for clarity.
- Enhanced main loop in app.py to handle exit signals and exceptions.
- Added shutdown methods in Ingestor and DataManager classes for graceful resource cleanup.
- Updated unit tests to validate shutdown behavior and exception handling.
- Introduced coverage configuration to omit specific files.
- Changed logging statements in Ingestor and OpcManager from info to debug level to reduce log verbosity and improve performance.
- This adjustment aids in better log management while maintaining necessary traceability for debugging purposes.
- Added logging statements to OpcManager to track data change notifications and cycle count resets for monitored tags.
- Improved traceability of events related to OPC cycles, aiding in debugging and monitoring.
Enhance ingestor and manager classes with notification handling
- Integrated NotificationHandler into Ingestor, DataManager, IngestorManager, and OpcManager for improved error reporting and monitoring.
- Updated methods to send notifications on critical events such as Kafka publishing errors, OPC connection issues, and cycle count warnings.
- Refactored related tests to ensure coverage of new notification functionalities and validate integration with existing components.
- Improved logging and error handling across the system to enhance traceability and operational insights.
Refactor OpcManager initialization and unsubscribe method; enhance logging
- Removed the `initialize_from_config` method from `OpcManager` class and adjusted the constructor to handle configuration directly.
- Improved the `unsubscribe` method to include detailed logging for non-existent subscriptions.
- Updated tests in `test_opc_manager.py` to reflect changes in the `OpcManager` class.
- Commented out Docker-related fixtures in `conftest.py` for potential future use.
- Enhanced test coverage in `test_single_node.py` and `test_data_manager.py` with additional scenarios and assertions.
- Introduced new methods in `Ingestor` and `IngestorManager` classes to manage server subscriptions and leases more effectively.
- Added new tests for `Ingestor` class to validate initialization and slot management logic.
- Implemented logging improvements across various classes to ensure better traceability of actions and errors.
Add temporary functiontal tests, that must be removed soon.
Refactor tests: Remove outdated resource manager tests and add functional tests with Docker and Kafka integration
- Deleted existing unit tests for ResourceManager.
- Introduced new functional tests for single node operations with Redis and Kafka.
- Added Docker Compose setup for test environment.
- Implemented fixtures for Redis and Kafka consumers.
- Created comprehensive tests for data publishing and slot management.
- Added unit tests for DataManager and IngestorManager with mocked dependencies.
- Included tests for OpcManager covering connection, subscription, and data change notifications.
Enhance README and Docker setup; improve IngestorManager and DataManager functionality, add error handling, and implement unit tests for OPC initialization and subscription management.
Implement initial structure for data ingestion with Kafka and OPC UA integration
- Added DataManager for Kafka message handling
- Introduced IngestorManager to manage data ingestion processes
- Created OpcManager for OPC UA client interactions
- Developed ResourceManager for Redis-based resource management
- Established testing framework with unit tests for DataManager and OpcManager
- Configured project settings for Python testing with pytest
