GatewayRuntime builds and owns the F4 object graph (CertProvider/pool ->
WriteEngine/NatsWriteConsumer -> ActiveReadEngine -> optional GrpcServer),
reusing the existing ingestor's logger/notification_handler/metrics_controller
and DataManager (CertProvider shares its Mongo connection instead of opening
a second one). app.py starts it after prepare_ingestor, sweeps idle OPC
connections each loop iteration, and stops it on shutdown. A NATS connect
failure at startup is logged/notified but never crashes the app -- the read
path keeps working and idle_sweep() retries the consumer start.
ingestor.py needed no changes: data_manager is already reachable via
ingestor.ingestor_manager.data_manager.
Also documents the new MinIO/Postgres env vars this wiring reads (via
sientia_do's build_postgres_config for Postgres) in .env.example, and
updates test_app.py's fixture to mock GatewayConfig/GatewayRuntime so the
existing app tests keep exercising only what they mocked before.
IngestorManager.__del__ calls asyncio.run(self.shutdown()). Adding
tests/unit/connection/test_pool.py shifted GC timing enough that a
cyclic-referenced IngestorManager instance from an earlier test was
swept mid test___del__, double-calling the patched asyncio mock and
failing the assertion. Deterministic gc.collect() after each test in
this file keeps finalizers scoped to their own test.
Enhance release workflow to trigger only on merged pull requests and streamline data insertion in DataManager
- Updated the release workflow to execute only when a pull request is merged.
- Simplified the document insertion logic in DataManager by consolidating the dictionary construction.
- Refactored OPC Manager tests to utilize asynchronous mocks for improved accuracy in testing.
Enhance Ingestor Class with Asynchronous Metrics and Monitoring Integration
- Refactored Ingestor class to inherit from SientiaMonitoring for improved observability.
- Updated methods to utilize asynchronous operations for declaring active ingestors and managing slots.
- Integrated pod_id label into various metrics for better tracking.
- Adjusted unit tests to reflect changes in the Ingestor initialization and asynchronous behavior.
Refactor Ingestor and Metrics Classes for Enhanced Asynchronous Operations
- Updated IngestorManager, OpcManager, and ResourceManager to utilize asynchronous methods for improved performance.
- Integrated MetricsController into various classes for better observability and monitoring.
- Adjusted unit tests to accommodate the new asynchronous behavior, ensuring proper mocking of async methods.
- Removed deprecated Redis metrics and streamlined resource management logic.
SIENTIAPDE-1325: Refactor Ingestor and Manager Classes for Enhanced Asynchronous Operations
- Introduced asynchronous methods across Ingestor, IngestorManager, DataManager, and OpcManager to improve performance and responsiveness.
- Integrated MetricsController into various classes for better observability and monitoring.
- Updated Redis and MongoDB interactions to support asynchronous operations, enhancing data handling efficiency.
- Removed deprecated Redis metrics and streamlined resource management logic.
- Adjusted unit tests to accommodate the new asynchronous behavior and ensure proper mocking of async methods.
Refactor IngestorManager and OpcManager for improved server management
- Introduced a new method `remove_server` in IngestorManager to encapsulate server removal logic, enhancing code clarity and reusability.
- Updated `update_opc_servers` to utilize the new `remove_server` method for better maintainability.
- Refactored security settings in OpcManager to use direct attribute assignments instead of method calls, streamlining the code.
- Adjusted unit tests to reflect changes in the IngestorManager and OpcManager, ensuring proper asynchronous behavior and mocking.
Updated the Ingestor and IngestorManager classes to use async methods for checking OPC server integrity and shutting down OPC managers. Adjusted related tests to ensure proper mocking of asynchronous behavior.
Update image tag in values.yaml and refactor OpcManager error handling
- Updated the image tag in values.yaml from "0.4.5" to "0.4.9" for the latest version.
- Refactored error handling in OpcManager to improve logging and maintain code clarity.
- Adjusted unit tests for better readability and consistency in assertions.
Refactor OpcManager connection error handling and enhance unit tests
- Simplified error handling during connection attempts in OpcManager by removing redundant disconnection logic.
- Updated unit tests to include subscription_period_ms in server configuration and adjusted connection assertions to include a timeout parameter.
- Added new tests for disconnection fallback functionality to ensure robust error handling during disconnect attempts.
Enhance OpcManager to accept subscription period from configuration
- Updated OpcManager to include subscription_period_ms as a parameter for better configurability.
- Refactored create_subscription method to utilize the new subscription period parameter.
- Adjusted unit tests to reflect changes in subscription handling and ensure correct functionality.
Refactor app.py and enhance unit tests for better clarity and structure
- Updated `app.py` to improve async compatibility with a clearer comment.
- Added blank lines in `test_ingestor.py` for improved readability and organization of test cases.
Refactor async handling and enhance unit tests for ingestor manager
- Updated `app.py` to improve async compatibility by adding a comment for clarity.
- Added new unit tests to handle scenarios where the ingestor manager is None, ensuring robustness in `manage_leases` and `loop` methods.
- Enhanced existing tests to cover edge cases related to the ingestor manager's state.
Refactor project structure and update configurations
- Deleted the empty `__init__.py` file to clean up the project structure.
- Renamed the project from "laborious" to "ingestor" in `pyproject.toml`, updating the description accordingly.
- Improved type hints and conditional checks in the `Ingestor`, `DataManager`, `IngestorManager`, and `OpcManager` classes for better code clarity and type safety.
- Enhanced error handling and assertions in various methods to ensure robustness.
- Updated unit tests to reflect changes in class names and error handling improvements.
- Updated the `sientia-dataops-library` dependency version from 1.4.3 to 1.4.6 in `requirements.txt`.
- Modified the GitHub Actions workflow to install development and runtime dependencies separately, improving clarity and organization.
- Added code formatting and linting checks using Ruff, along with type checking using mypy, to ensure code quality.
- Updated `.gitignore` to include additional cache directories and log files.
- Refactored code in various files for consistency in string formatting and improved logging messages.
Enhance unit tests for async shutdown and improve assertions
- Updated `test_app.py` to verify multiple calls to `asyncio_sleep` and ensure proper handling of sleep intervals.
- Added new tests in `test_ingestor.py` and `test_opc_manager.py` to validate shutdown behavior and error handling for async operations.
- Improved assertions in existing tests to enhance reliability and clarity of test outcomes.
