Add unique constraint on (model_id, timestamp, variable) in schema and implement idempotent export test to ensure no duplicates are created. Update CoreScouter and related tests to handle conflict resolution by ignoring duplicates.
Enhance API class to format timestamps correctly and improve logging. Update test cases to include 'end_time' parameter for better data retrieval flexibility.
Update tests.ipynb and API class in scouter module
- Reset execution counts in tests.ipynb for reproducibility.
- Removed unnecessary outputs and adjusted execution counts for clarity.
- Enhanced API class to accept 'end_time' parameter for improved data retrieval flexibility.
- Commented out timestamp formatting and normalization for future adjustments.
Refactor API class to remove unnecessary blank line and improve code readability. Update test cases to rename 'timeout' parameter to 'request_timeout' for consistency.
Update tests.ipynb and prepare_worker.py
- Reset execution counts in tests.ipynb for reproducibility.
- Removed error outputs and added new tags in TAG_NAMES for enhanced data retrieval.
- Introduced JSON export of web_ids to 'web_ids.json' for better data management.
- Added comments in prepare_worker.py to clarify worker configuration parameters.
Enhance PI Web API Scouter Workflow and Documentation
- Added support for PI Web API data ingestion, including real-time and historical data retrieval.
- Implemented timestamp normalization to ensure consistency across records.
- Updated README.md to reflect new features and detailed workflow execution flow.
- Enhanced API class with comprehensive error handling and cleanup operations.
- Improved CoreScouter workflow with early exit conditions and detailed processing steps.
- Updated configuration parameters for better clarity and usability.
Refactor metrics and imports across multiple files
- Removed unused import of CORE_LABELS in metrics.py.
- Updated import statements in api.py for consistency.
- Cleaned up import order in worker.py for better readability.
- Added missing newline at the end of metrics.py.
- Adjusted formatting in core_scouter.py to ensure proper syntax.
- Removed redundant 'on_conflict' and 'unique_columns' keys from test cases in test_core_scouter.py for clarity.
Update pytest_asyncio fixture scopes in conftest.py for improved test isolation and add asyncio_default_fixture_loop_scope in pyproject.toml. Remove outdated scenarios from scenarios.md and delete unused test files for cleaner codebase.
Refactor PIWebAPIClient and Update Configuration Imports
- Removed outdated PostgreSQL, Redis, MongoDB, and API configuration functions from connectors_config.py.
- Deleted the pi_web_api_client.py file as part of the refactor.
- Updated imports in worker.py and api.py to use the new repository structure.
- Cleaned up scenarios.md by removing obsolete scenarios related to unique constraint violations.
- Commented out the specific version of the sientia-dataops-library in requirements.txt for flexibility.
Update pyproject.toml to enable automatic asyncio mode, modify pytest_asyncio fixture scopes in conftest.py for better isolation, and streamline e2e scenarios documentation in scenarios.md by removing outdated scenarios and reorganizing sections for clarity.
Update requirements-dev.txt to add E2E testing dependencies: fakeredis and mongomock for in-memory testing, and include testcontainers for PostgreSQL support.
Invert startTime and endTime parameters in PIWebAPIClient to ensure descending order results. Update tests to verify parameter handling and use pytest.approx for floating-point comparisons.
Update test_api.py and test_core_scouter.py to enhance timestamp validation and add new test cases
- Updated timestamp assertions in test_get_tag_values_success to reflect the correct values.
- Added new test cases in test_core_scouter.py to handle scenarios with zero affected rows and without debug data package, ensuring proper workflow execution and activity method assertions.
Refactor condition in core_scouter.py for improved readability
- Simplified the condition checking for affected rows in the data export logic by removing unnecessary whitespace.
Update tests.ipynb to improve execution count management and enhance data handling
- Adjusted execution counts for various code cells to maintain consistent state.
- Added error handling for KeyError in API response processing.
- Updated data processing logic to include timestamp formatting and CSV export functionality.
Update requirements.txt and core_scouter.py
- Commented out the Git repository reference for sientia-dataops-library in requirements.txt and replaced it with a local path.
- Modified the condition for data export in core_scouter.py to check for affected rows, enhancing data handling logic.
Update requirements.txt to change sientia-dataops-library dependency from a local path to a Git repository reference at version 1.7.0 for improved dependency management.
Update requirements and enhance tests.ipynb functionality
- Changed the dependency path for sientia-dataops-library in requirements.txt to a local directory.
- Updated execution counts in tests.ipynb for better state management and added new code cells for retrieving web IDs from the PI Web API, improving data handling capabilities.
