Add unique constraint on (model_id, timestamp, variable) in schema and implement idempotent export test to ensure no duplicates are created. Update CoreScouter and related tests to handle conflict resolution by ignoring duplicates.
Update project configuration and dependencies
- Added .mypy_cache and .cursor to .gitignore.
- Changed asyncio_default_fixture_loop_scope and asyncio_default_test_loop_scope to "session" in pyproject.toml.
- Updated e2e testing dependencies in requirements-dev.txt, replacing fakeredis and mongomock with pytest-httpserver.
- Updated requirements.txt to use sientia_do instead of a specific git commit.
- Modified sonar-project.properties to remove a file from coverage exclusions.
- Enhanced E2E test fixtures in e2e/conftest.py for better container management.
- Cleaned up e2e test files related to CoreScouter and PIWebAPIScouter workflows.
Enhance API class to format timestamps correctly and improve logging. Update test cases to include 'end_time' parameter for better data retrieval flexibility.
Update tests.ipynb and API class in scouter module
- Reset execution counts in tests.ipynb for reproducibility.
- Removed unnecessary outputs and adjusted execution counts for clarity.
- Enhanced API class to accept 'end_time' parameter for improved data retrieval flexibility.
- Commented out timestamp formatting and normalization for future adjustments.
Refactor API class to remove unnecessary blank line and improve code readability. Update test cases to rename 'timeout' parameter to 'request_timeout' for consistency.
Update tests.ipynb and prepare_worker.py
- Reset execution counts in tests.ipynb for reproducibility.
- Removed error outputs and added new tags in TAG_NAMES for enhanced data retrieval.
- Introduced JSON export of web_ids to 'web_ids.json' for better data management.
- Added comments in prepare_worker.py to clarify worker configuration parameters.
Enhance PI Web API Scouter Workflow and Documentation
- Added support for PI Web API data ingestion, including real-time and historical data retrieval.
- Implemented timestamp normalization to ensure consistency across records.
- Updated README.md to reflect new features and detailed workflow execution flow.
- Enhanced API class with comprehensive error handling and cleanup operations.
- Improved CoreScouter workflow with early exit conditions and detailed processing steps.
- Updated configuration parameters for better clarity and usability.
Refactor metrics and imports across multiple files
- Removed unused import of CORE_LABELS in metrics.py.
- Updated import statements in api.py for consistency.
- Cleaned up import order in worker.py for better readability.
- Added missing newline at the end of metrics.py.
- Adjusted formatting in core_scouter.py to ensure proper syntax.
- Removed redundant 'on_conflict' and 'unique_columns' keys from test cases in test_core_scouter.py for clarity.
Refactor PIWebAPIClient and Update Configuration Imports
- Removed outdated PostgreSQL, Redis, MongoDB, and API configuration functions from connectors_config.py.
- Deleted the pi_web_api_client.py file as part of the refactor.
- Updated imports in worker.py and api.py to use the new repository structure.
- Cleaned up scenarios.md by removing obsolete scenarios related to unique constraint violations.
- Commented out the specific version of the sientia-dataops-library in requirements.txt for flexibility.
Invert startTime and endTime parameters in PIWebAPIClient to ensure descending order results. Update tests to verify parameter handling and use pytest.approx for floating-point comparisons.
Refactor condition in core_scouter.py for improved readability
- Simplified the condition checking for affected rows in the data export logic by removing unnecessary whitespace.
Update requirements.txt and core_scouter.py
- Commented out the Git repository reference for sientia-dataops-library in requirements.txt and replaced it with a local path.
- Modified the condition for data export in core_scouter.py to check for affected rows, enhancing data handling logic.
Update requirements and enhance tests.ipynb functionality
- Changed the dependency path for sientia-dataops-library in requirements.txt to a local directory.
- Updated execution counts in tests.ipynb for better state management and added new code cells for retrieving web IDs from the PI Web API, improving data handling capabilities.
Refactor imports and enhance timestamp handling in API and test files
- Removed redundant import statements in api.py and prepare_worker.py for cleaner code.
- Updated timestamp format in test cases to include timezone information for consistency.
- Adjusted test configurations to align with new API structure for improved accuracy in data handling.
Enhance PI Web API integration and update test configurations
- Adjusted execution counts in tests.ipynb for better notebook state management.
- Introduced a new pi_web_api_query structure in tests.ipynb for improved API configuration.
- Updated API class to format timestamps using DATETIME_FORMAT_WITH_TZ.
- Refactored PIWebAPIClient to ensure correct web ID retrieval and emit metrics accurately.
- Streamlined workflow parameters in pi_web_api_scouter.py for clarity and flexibility.
SIENTIAPDE-1445 Add PI Web API configuration to .env.example and enhance worker setup
- Introduced new environment variables for PI Web API configuration in .env.example.
- Updated prepare_worker.py to utilize a camelCase to kebab-case conversion function for queue naming.
- Enhanced logging in prepare_worker to provide better insights during worker preparation.
- Adjusted execution counts in tests.ipynb for improved notebook state management.
Refactor workflow parameters in tests.ipynb and prepare_worker.py to enhance clarity and flexibility. Updated workflow_id and task_queue to accept dynamic values, and standardized timeout parameters for improved configuration.
Enhance Activities and API Integration
- Updated Activities class to include API operations for external data ingestion.
- Added API configuration builder to connectors_config.py for environment variable management.
- Integrated API configuration into worker setup.
- Expanded unit tests to cover new API functionality and configuration handling.
- Updated requirements.txt to include pycurl and prometheus-client for enhanced metrics support.
Refactor prepare_worker.py to improve type annotations and code clarity. Update import statements and enhance formatting for better readability. Adjust comments in worker.py for consistency.
SIENTIAPDE-1441: Refactor worker configuration in worker.py to utilize a new prepare_worker function for cleaner setup. Update environment variable names in values.yaml for consistency and clarity, adjusting memory requests and enhancing health check commands for improved application monitoring.
SIENTIAPDE-1442: Enhance Temporal worker configuration by adding environment variables for concurrency limits and poller behavior settings in values.yaml and worker.py for optimized performance.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1273, increment Helm chart version to 0.6.0, and modify metrics registration in Gates class to skip None values. Update test cases to reflect changes in tag values.
Enhance Redis activity by formatting set and get operations for improved readability and consistency. Update tests to include metadata in assertions for better verification of Redis interactions.
Refactor notification methods in Gates, MongoDB, and Redis classes to use asynchronous send_notification_async. Update related tests to ensure proper async handling and verification of notifications.
Refactor initialization of Activities, Gates, and MongoDB classes for improved readability by using multi-line argument formatting. Update related tests to match the new initialization style.
Update environment configuration and refactor activities to include metrics controller. Remove Kafka settings and adjust Redis and MongoDB initialization. Update tests to reflect changes in initialization and metrics tracking.
Update GITHUB_BRANCH in values.yaml to feature/SIENTIAPDE-1325 for specific external operation metrics. Refactor shutdown methods in Activities class to ensure proper closure of MongoDB, Redis, and Gates connections. Adjust tests to reflect these changes and enhance repository pattern implementation for MongoDB and Redis activities.
Refactor MongoDB, Redis, and Gates classes to extend SientiaMonitoring instead of BaseActivity. Update requirements to point to local dataops library path. Implement repository pattern for MongoDB and Redis operations, enhancing code organization and maintainability.
SIENTIAPDE-1314: Update workflow name from 'sub_core_scouter' to 'subworkflow.core_scouter' in scouter.py and core_scouter.py, and adjust test references accordingly in test_scouter.py for consistency.
