Enhance documentation and improve shutdown methods for DataManager and IngestorManager
- Updated README.md to clarify persistent storage capabilities.
- Enhanced docstrings in DataManager and IngestorManager for shutdown processes, detailing resource cleanup and error handling.
- Improved comments in various methods to provide clearer explanations of functionality and side effects.
Update README.md to clarify data streaming and storage features
- Revised descriptions for Real-time Data Streaming and Persistent Storage to emphasize historical data streaming capabilities.
- Updated Data Manager section to reflect the integration of both MongoDB and Kafka for data management.
- Enhanced configuration example in the prerequisites section with additional fields for server configuration.
Remove deprecated files and enhance documentation
- Deleted `coverage.sh`, `docker-compose.yaml`, `Dockerfile`, and simulator-related files to streamline the project structure.
- Updated `README.md` to provide a comprehensive overview of the OPC Ingestor, including features, architecture, installation, usage, and troubleshooting.
- Enhanced docstrings across various classes and methods in the `ingestor` module for better clarity and maintainability.
- Improved Prometheus metrics documentation in `metrics.py` to ensure proper monitoring and observability of the system.
Enhance unit tests for async shutdown and improve assertions
- Updated `test_app.py` to verify multiple calls to `asyncio_sleep` and ensure proper handling of sleep intervals.
- Added new tests in `test_ingestor.py` and `test_opc_manager.py` to validate shutdown behavior and error handling for async operations.
- Improved assertions in existing tests to enhance reliability and clarity of test outcomes.
Refactor tests for async compatibility and enhance IngestorManager functionality
- Updated test cases in `test_app.py`, `test_ingestor.py`, and `test_ingestor_manager.py` to use async/await syntax for improved concurrency.
- Refactored methods in IngestorManager and related classes to support asynchronous operations, ensuring non-blocking behavior during execution.
- Enhanced mock setups in tests to accommodate async methods, improving test reliability and performance.
Update values.yaml and refactor shutdown methods for improved configuration and async handling
- Changed replica count from 2 to 1 in values.yaml for deployment optimization.
- Updated image tag from 0.4.3 to 0.4.5 in values.yaml for version consistency.
- Refactored shutdown method in IngestorManager to use await server.shutdown() for better async operation.
- Removed __del__ method in OpcManager to prevent blocking during cleanup.
Refactor Ingestor and OPC Manager for Asynchronous Operations
- Updated main function to be asynchronous and integrated asyncio for better concurrency.
- Refactored Ingestor methods to support async operations, including prepare_ingestor, loop, and shutdown.
- Enhanced IngestorManager and OpcManager with async methods for improved performance and responsiveness.
- Replaced blocking calls with await statements to ensure non-blocking behavior during operations.
- Added a new run_async_main function to handle the async event loop setup.
SIENTIAPDE-1193 Update values.yaml to include new environment variables for enhanced configuration
- Added new environment variables for improved deployment flexibility.
- Updated existing configurations to align with recent changes in the application structure.
Update requirements and refactor timestamp handling in DataManager
- Updated sientia-dataops-library version to 1.4.3 in requirements.txt.
- Refactored timestamp insertion in DataManager to use the now() function for improved consistency.
Update requirements and refactor datetime handling in OpcManager for timezone support
- Commented out the specific version of sientia-dataops-library in requirements.txt and added a local path.
- Updated GITHUB_BRANCH in values.yaml for consistency with current development.
- Refactored source_timestamp handling in OpcManager to include timezone information.
- Adjusted timestamp formatting in test cases to reflect the new datetime format with timezone.
Refactor logging levels in Ingestor and IngestorManager for improved clarity
- Changed debug log statements to info level in Ingestor class for better visibility of subscription and management actions.
- Updated log level for initializing OPC manager in IngestorManager to info for consistent logging practices.
Update Ingestor project name and metadata keys to lowercase for consistency
- Changed project_name, workflow_name, and schema_name from uppercase to lowercase in Ingestor class and corresponding unit tests.
- Enabled metrics service with updated configuration for Prometheus ServiceMonitor.
- Removed disabled API and OPC service configurations to streamline values.yaml.
- Updated helm upgrade command version to 0.5.0.
- Updated sientia-dataops-library version to 1.4.1 in requirements.txt.
- Changed image tag from 0.3.2 to 0.4.2 in values.yaml.
- Refactored logger imports from sientia_do.temporal.utils.logger to sientia_do.observability.logger in multiple manager files for improved observability.
Refactor test assertions in test_opc_manager.py for improved readability
- Updated the assertion syntax for notification calls in the test_check_opc_listenning_error_notification_and_retry function to use dictionary literals for clarity.
Update requirements and refactor manager classes to remove pod_id
- Updated the sientia-dataops-library dependency version to 1.3.8 in requirements.txt.
- Refactored IngestorManager, OpcManager, and ResourceManager classes to remove pod_id from their initialization and internal handling, enhancing code clarity and consistency.
- Adjusted unit tests to reflect the removal of pod_id, ensuring they remain functional and accurate.
Update BaseActivity initialization in manager classes to enable error counter metric
- Modified the initialization of DataManager, IngestorManager, OpcManager, and ResourceManager to include 'set_error_counter=True' for enhanced error tracking.
Update BaseActivity initialization in manager classes to disable error counter metric
- Modified the initialization of DataManager, IngestorManager, OpcManager, and ResourceManager to include 'set_error_counter_metric=False' for improved error handling.
Refactor unit tests to improve readability and maintainability
- Updated test assertions and mock setups in test_app.py for clarity.
- Added 'pod_id' to the metadata in test_ingestor.py for consistency.
- Introduced a new test for TAG_WRITTEN_COUNT in test_metrics.py to verify its definition.
- Enhanced test data structure in test_data_manager.py to include additional fields.
SIENTIAPDE-1174 Add pod_id to Ingestor metadata for improved tracking
- Included 'pod_id' in the metadata dictionary of the Ingestor class to enhance tracking capabilities.
Update values.yaml to modify replica count and GITHUB_BRANCH environment variable
- Increased replicaCount from 1 to 3 for improved scalability.
- Updated GITHUB_BRANCH environment variable to SIENTIAPDE-1174-mapear-e-implementar-metricas-a-serem-criadas.
Update requirements and environment configurations
- Changed the sientia-dataops-library dependency to a relative path in requirements.txt.
- Updated image tag to 0.3.0 in values.yaml.
- Renamed HTTP_SERVER_PORT to HTTP_METRICS_PORT in app.py and updated its default value to 9090.
- Added TAG_WRITTEN_COUNT metric in metrics.py to track writing processes.
- Incremented TAG_WRITTEN_COUNT in DataManager upon successful data insertion.
Update dependencies and environment variables
- Updated sientia-dataops-library dependency version to 1.3.5 in requirements.txt.
- Changed image tag to 0.2.7 in values.yaml.
- Updated GITHUB_BRANCH environment variable to SIENTIAPDE-1172-criar-pipeline-de-alertas-orquestrador in values.yaml.
Update values.yaml to add metrics service configuration
- Introduced a new 'metrics' service with ClusterIP type, port 9090, and targetPort 9090.
- Updated the serviceMonitor port to reference the new 'metrics' service.
- Changed HTTP_SERVER_PORT environment variable value to 9090 to align with the new metrics service.
SIENTIAPDE-1163
Refactor Ingestor and DataManager classes to enhance metadata handling
- Updated Ingestor class to include additional metadata fields: 'workflow_name' and 'schema_name'.
- Modified DataManager to accept metadata during initialization.
- Adjusted unit tests to validate the new metadata structure and ensure proper functionality across various managers.
- Removed unused logger initialization in tests for cleaner code.
