Review feedback (Vitor): gating the batch-max normalization behind
max_count == 1 just hid the same collapsing behavior for the single-point
case, where it's redundant anyway (aggregation/export don't need it) while
still being the wrong fix conceptually. Removed the normalization block
entirely so every record always keeps its own real timestamp from the PI
Web API, regardless of max_count.
Updated unit tests accordingly and confirmed no e2e scenario needs
adjustment: aggregate_data groups by (tag, name) and always collapses to
one output row per tag per execution, independent of per-record timestamp
values, so pi_web_api_scouter_multiple_tags' aggregated-value assertions
are unaffected.
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Timestamp normalization in API.get_tag_values previously overwrote all
records with the batch's maximum timestamp regardless of max_count.
With max_count > 1, this collapsed distinct historical points for the
same tag onto a single timestamp, causing them to collide on the
(model_id, timestamp, variable) unique constraint and get silently
dropped via ON CONFLICT DO NOTHING.
Normalization is now only applied when max_count == 1, which is the
only case where it is meaningful (aligning minor sampling differences
across tags for a single point each).
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Update project configuration and dependencies
- Added .mypy_cache and .cursor to .gitignore.
- Changed asyncio_default_fixture_loop_scope and asyncio_default_test_loop_scope to "session" in pyproject.toml.
- Updated e2e testing dependencies in requirements-dev.txt, replacing fakeredis and mongomock with pytest-httpserver.
- Updated requirements.txt to use sientia_do instead of a specific git commit.
- Modified sonar-project.properties to remove a file from coverage exclusions.
- Enhanced E2E test fixtures in e2e/conftest.py for better container management.
- Cleaned up e2e test files related to CoreScouter and PIWebAPIScouter workflows.
Enhance API class to format timestamps correctly and improve logging. Update test cases to include 'end_time' parameter for better data retrieval flexibility.
Refactor API class to remove unnecessary blank line and improve code readability. Update test cases to rename 'timeout' parameter to 'request_timeout' for consistency.
Invert startTime and endTime parameters in PIWebAPIClient to ensure descending order results. Update tests to verify parameter handling and use pytest.approx for floating-point comparisons.
Update test_api.py and test_core_scouter.py to enhance timestamp validation and add new test cases
- Updated timestamp assertions in test_get_tag_values_success to reflect the correct values.
- Added new test cases in test_core_scouter.py to handle scenarios with zero affected rows and without debug data package, ensuring proper workflow execution and activity method assertions.
Refactor imports and enhance timestamp handling in API and test files
- Removed redundant import statements in api.py and prepare_worker.py for cleaner code.
- Updated timestamp format in test cases to include timezone information for consistency.
- Adjusted test configurations to align with new API structure for improved accuracy in data handling.
Enhance Activities and API Integration
- Updated Activities class to include API operations for external data ingestion.
- Added API configuration builder to connectors_config.py for environment variable management.
- Integrated API configuration into worker setup.
- Expanded unit tests to cover new API functionality and configuration handling.
- Updated requirements.txt to include pycurl and prometheus-client for enhanced metrics support.
