Review feedback (Vitor): gating the batch-max normalization behind
max_count == 1 just hid the same collapsing behavior for the single-point
case, where it's redundant anyway (aggregation/export don't need it) while
still being the wrong fix conceptually. Removed the normalization block
entirely so every record always keeps its own real timestamp from the PI
Web API, regardless of max_count.
Updated unit tests accordingly and confirmed no e2e scenario needs
adjustment: aggregate_data groups by (tag, name) and always collapses to
one output row per tag per execution, independent of per-record timestamp
values, so pi_web_api_scouter_multiple_tags' aggregated-value assertions
are unaffected.
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Timestamp normalization in API.get_tag_values previously overwrote all
records with the batch's maximum timestamp regardless of max_count.
With max_count > 1, this collapsed distinct historical points for the
same tag onto a single timestamp, causing them to collide on the
(model_id, timestamp, variable) unique constraint and get silently
dropped via ON CONFLICT DO NOTHING.
Normalization is now only applied when max_count == 1, which is the
only case where it is meaningful (aligning minor sampling differences
across tags for a single point each).
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Update project configuration and dependencies
- Added .mypy_cache and .cursor to .gitignore.
- Changed asyncio_default_fixture_loop_scope and asyncio_default_test_loop_scope to "session" in pyproject.toml.
- Updated e2e testing dependencies in requirements-dev.txt, replacing fakeredis and mongomock with pytest-httpserver.
- Updated requirements.txt to use sientia_do instead of a specific git commit.
- Modified sonar-project.properties to remove a file from coverage exclusions.
- Enhanced E2E test fixtures in e2e/conftest.py for better container management.
- Cleaned up e2e test files related to CoreScouter and PIWebAPIScouter workflows.
Enhance API class to format timestamps correctly and improve logging. Update test cases to include 'end_time' parameter for better data retrieval flexibility.
Update tests.ipynb and API class in scouter module
- Reset execution counts in tests.ipynb for reproducibility.
- Removed unnecessary outputs and adjusted execution counts for clarity.
- Enhanced API class to accept 'end_time' parameter for improved data retrieval flexibility.
- Commented out timestamp formatting and normalization for future adjustments.
Refactor API class to remove unnecessary blank line and improve code readability. Update test cases to rename 'timeout' parameter to 'request_timeout' for consistency.
Enhance PI Web API Scouter Workflow and Documentation
- Added support for PI Web API data ingestion, including real-time and historical data retrieval.
- Implemented timestamp normalization to ensure consistency across records.
- Updated README.md to reflect new features and detailed workflow execution flow.
- Enhanced API class with comprehensive error handling and cleanup operations.
- Improved CoreScouter workflow with early exit conditions and detailed processing steps.
- Updated configuration parameters for better clarity and usability.
Refactor metrics and imports across multiple files
- Removed unused import of CORE_LABELS in metrics.py.
- Updated import statements in api.py for consistency.
- Cleaned up import order in worker.py for better readability.
- Added missing newline at the end of metrics.py.
- Adjusted formatting in core_scouter.py to ensure proper syntax.
- Removed redundant 'on_conflict' and 'unique_columns' keys from test cases in test_core_scouter.py for clarity.
Refactor PIWebAPIClient and Update Configuration Imports
- Removed outdated PostgreSQL, Redis, MongoDB, and API configuration functions from connectors_config.py.
- Deleted the pi_web_api_client.py file as part of the refactor.
- Updated imports in worker.py and api.py to use the new repository structure.
- Cleaned up scenarios.md by removing obsolete scenarios related to unique constraint violations.
- Commented out the specific version of the sientia-dataops-library in requirements.txt for flexibility.
Refactor imports and enhance timestamp handling in API and test files
- Removed redundant import statements in api.py and prepare_worker.py for cleaner code.
- Updated timestamp format in test cases to include timezone information for consistency.
- Adjusted test configurations to align with new API structure for improved accuracy in data handling.
Enhance PI Web API integration and update test configurations
- Adjusted execution counts in tests.ipynb for better notebook state management.
- Introduced a new pi_web_api_query structure in tests.ipynb for improved API configuration.
- Updated API class to format timestamps using DATETIME_FORMAT_WITH_TZ.
- Refactored PIWebAPIClient to ensure correct web ID retrieval and emit metrics accurately.
- Streamlined workflow parameters in pi_web_api_scouter.py for clarity and flexibility.
Enhance Activities and API Integration
- Updated Activities class to include API operations for external data ingestion.
- Added API configuration builder to connectors_config.py for environment variable management.
- Integrated API configuration into worker setup.
- Expanded unit tests to cover new API functionality and configuration handling.
- Updated requirements.txt to include pycurl and prometheus-client for enhanced metrics support.
