sientia_do 1.12.5 pins the psycopg2 driver in the Postgres engine URL.
Without it, rebuilding the image pulls SQLAlchemy 2.1.x, whose default
'postgresql://' driver is psycopg (v3), and the worker crashes at
startup with "ModuleNotFoundError: No module named 'psycopg'".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Feedback (Vitor): the intent was a single scenario covering both
max_count=1 and max_count>1, checking that in both cases each tag's
own latest (properly aggregated) timestamp prevails — not just the
max_count>1 case from the previous commit.
Renamed the existing fixture to
pi_web_api_scouter_distinct_tag_timestamps_multi_point.json and added
a sibling _single_point.json (max_count=1, one point per tag, distinct
timestamps across tags). test_scenario_2_1_4 is now parametrized over
both fixtures with the same assertion: each tag's stored Postgres row
keeps its own real timestamp, never another tag's.
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Existing e2e coverage (2.1.2 multiple_tags) never exercised the actual
bug, since all its tags happened to share the same latest timestamp,
so aggregation's own per-tag last-timestamp pick coincided with the
old (buggy) batch-wide max regardless of the fix.
Add scenario 2.1.4: two tags with max_count=2, each with two historical
points, but with genuinely different latest timestamps (~12:00 vs
~15:00). Asserts each tag's stored Postgres row keeps its own real
timestamp rather than the other tag's. This test fails against the
pre-fix code (both rows would show ~15:00) and passes with the fix.
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Review feedback (Vitor): gating the batch-max normalization behind
max_count == 1 just hid the same collapsing behavior for the single-point
case, where it's redundant anyway (aggregation/export don't need it) while
still being the wrong fix conceptually. Removed the normalization block
entirely so every record always keeps its own real timestamp from the PI
Web API, regardless of max_count.
Updated unit tests accordingly and confirmed no e2e scenario needs
adjustment: aggregate_data groups by (tag, name) and always collapses to
one output row per tag per execution, independent of per-record timestamp
values, so pi_web_api_scouter_multiple_tags' aggregated-value assertions
are unaffected.
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Timestamp normalization in API.get_tag_values previously overwrote all
records with the batch's maximum timestamp regardless of max_count.
With max_count > 1, this collapsed distinct historical points for the
same tag onto a single timestamp, causing them to collide on the
(model_id, timestamp, variable) unique constraint and get silently
dropped via ON CONFLICT DO NOTHING.
Normalization is now only applied when max_count == 1, which is the
only case where it is meaningful (aligning minor sampling differences
across tags for a single point each).
SIENTIAPDE-1993
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses review feedback: requirements-local.txt was bumped to 1.12.2
but requirements.txt was left with an unpinned sientia_do, missing the
minimum version bump.
Add unique constraint on (model_id, timestamp, variable) in schema and implement idempotent export test to ensure no duplicates are created. Update CoreScouter and related tests to handle conflict resolution by ignoring duplicates.
Update PostgreSQL configuration in values.yaml
- Changed POSTGRES_USER from "sientia" to "postgres".
- Updated POSTGRES_PASSWORD to a new secure value.
Update project configuration and dependencies
- Added .mypy_cache and .cursor to .gitignore.
- Changed asyncio_default_fixture_loop_scope and asyncio_default_test_loop_scope to "session" in pyproject.toml.
- Updated e2e testing dependencies in requirements-dev.txt, replacing fakeredis and mongomock with pytest-httpserver.
- Updated requirements.txt to use sientia_do instead of a specific git commit.
- Modified sonar-project.properties to remove a file from coverage exclusions.
- Enhanced E2E test fixtures in e2e/conftest.py for better container management.
- Cleaned up e2e test files related to CoreScouter and PIWebAPIScouter workflows.
Add unique constraint on (model_id, timestamp, variable) in schema and implement idempotent export test to ensure no duplicates are created. Update CoreScouter and related tests to handle conflict resolution by ignoring duplicates.
Enhance API class to format timestamps correctly and improve logging. Update test cases to include 'end_time' parameter for better data retrieval flexibility.
Update tests.ipynb and API class in scouter module
- Reset execution counts in tests.ipynb for reproducibility.
- Removed unnecessary outputs and adjusted execution counts for clarity.
- Enhanced API class to accept 'end_time' parameter for improved data retrieval flexibility.
- Commented out timestamp formatting and normalization for future adjustments.
Refactor API class to remove unnecessary blank line and improve code readability. Update test cases to rename 'timeout' parameter to 'request_timeout' for consistency.
Update tests.ipynb and prepare_worker.py
- Reset execution counts in tests.ipynb for reproducibility.
- Removed error outputs and added new tags in TAG_NAMES for enhanced data retrieval.
- Introduced JSON export of web_ids to 'web_ids.json' for better data management.
- Added comments in prepare_worker.py to clarify worker configuration parameters.
