SonarQube on PR #45 reported 3 new issues; all fixed without behavior change:
- S3776 gather_read_tags (cognitive complexity 22): the merge of one resolved
read tag into its (server, address) entry moves to _add_read_tag (now 14).
- S3776 Orchestrator.run (16): the tag database load moves to
Orchestrator._load_tag_database (run back to 14). workflow.patched() is still
evaluated first on every execution, so its marker is always recorded.
- S1192 '$match' duplicated in the workflow module: the connector_tags tag_id
filter moves to orchestrator_functions.filter_by_tag_ids, which also returns a
new aggregation list.
194 unit tests (new: filter_by_tag_ids); real workflow on a local Temporal test
server still passes the 3 scenarios; the dev dry run is still equivalent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review (code-review skill, xhigh) of efa0d94: 8 findings, 6 fixed here,
2 documented (decisions for the team).
Fixed:
- The tag database is read only when a scouter read tag carries a tag_id, and
only the referenced tags ($match tag_id $in, int and text forms), through
aggregations that project only the fields the slot uses. Environments without
a tag database, or whose pipelines do not use tag_id, never query it (P4), and
other attributes (e.g. an ObjectId, which the Temporal JSON converter rejects)
never reach a payload. Fresh aggregation lists are passed, because
aggregate_documents_in_mongodb appends a stage to the list it gets.
- Incomplete connectors / connector_tags documents are left out of the indexes
(their read tags are notified as not found) instead of raising KeyError and
failing the whole run.
- frequency is stored as int, so 15000 vs '15000' across pipelines no longer
makes the slot depend on the pipeline order (O6).
- A missing, null or empty tag_id means "no tag database": an empty string no
longer drops a legacy read tag.
- A read tag without an integer frequency (or, without tag_id, without
server_id / tag_address / tag_name) is skipped and notified
(ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION) instead of failing
the slot build.
- Server slot keys can no longer collide (name:server_id is extended until free).
Documented in the README:
- The gateway's plan B (READ_ENGINE=legacy) reads v1 only: switching to it after
this deploy also requires rolling the Orchestrator back.
- Renaming a tag in connector_tags does not redeploy the scouter (schedules
update on the pipeline's updated_at).
Validation: 193 unit tests; the real workflow on a local Temporal test server
(sandbox on) with the real Formatters activities and the dev data: slot v2
written without querying the tag database, a second run inserts nothing, and a
tag_id pipeline queries both aggregations with the $in filter; the dev dry run
is still equivalent to the current v1 slot through the gateway's parse_slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Orchestrator now writes the slot v2 (contract C1) the OPC Gateway reads:
one entry per (server, address) with a names[] list, so an address read by N
pipelines is read once and written to N collections (fixes "one name per
address", where the second model's scouter discarded everything).
- O1: load connectors (opc_ua, Active) and connector_tags (Read, Active) in
parallel with the current loads. Queries are optional in the input with
defaults in code (D9); empty or missing collections load nothing. The loads
sit behind workflow.patched('SIENTIAPDE-1974-tag-database'), so a run in
flight during a deploy replays without them.
- O2: gather_read_tags groups by (server, address). A read tag with tag_id
resolves address/connector from connector_tags and the server from connectors
(RF3); without tag_id it keeps server_id + tag_address + opc-servers (RF4),
so every environment without a tag database keeps collecting (P4).
- O3/SC1: resolve_tag_names is the single source of tag names for both the
scouter's model_tags and the slot's names[].
- O4: build_tag_config emits server_id/name/url/server_uri/use_security/
subscription_period_ms + tags{frequency, active_read, names}; no certificate
paths (the gateway takes the certificate from MinIO). frequency = lowest
among pipelines; subscription period = connector value, else min/2 as before;
active_read off unless a pipeline enables it (lowest cycle_multiple, D3).
- O5: process_slots splits by address (names of an address never split) and
notifies, once per run, unknown tag_id, missing/inactive connector and
unknown opc-servers server_id.
- O6: create_slot_config writes only new or changed slots; the slot is built
in a stable order, so unchanged pipelines do not make the gateway
re-subscribe (and the "Inserted slots" report stops firing every run).
- O7: unit tests for multi-name, repeated address, unknown tag_id, inactive
connector, mixed tags, server-name clash, active_read, slot division,
stability, SC1 and the replay path (184 tests).
- O8: README documents the slot v2 contract and the new optional inputs.
- test_email.test_close: count only the instance under test; an Email left by
another test can be garbage-collected (Email.__del__ -> close) while the
patch is active, which the new tests' allocation pattern exposed.
Dry run with the dev data (scouter-sin-approx): the v2 slot is equivalent to
the current v1 slot as seen by the gateway's parse_slot (same server, addresses,
collections, names, frequency and subscription period).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the review-feedback fixes (batch-size default, requirements-local,
docstring trim) from PR #43 so #44 stays consistent with its base branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts:
# CLAUDE.md
- NOTIFICATION_QUEUE_BATCH_SIZE default reverted to 0 (unlimited) -- the
existing behavior before this ticket, so deployments must opt in to the
cap rather than having it silently applied at 5000
- requirements-local.txt: bump sientia_do pin to 1.12.3 to match requirements.txt
- Trim the load_latest_data docstring addition so it reads as documentation,
not a changelog entry
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
levels and model_ids (SIENTIAPDE-2149) both computed a per-group union with
identical "union only if every active group restricts, else unrestricted"
logic. Extracting _union_when_all_restrict removes the duplicated for/if
block for each axis, addressing the SonarQube quality gate failure on this
PR (cognitive complexity).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New, separate model_id-based filter distinct from the existing
ignore_models_id deny-list (SIENTIAPDE-2012): a group configuring
model_ids only receives notifications from those models. Both axes
apply together (AND) when a group sets both.
Design decisions made while implementing (matching the levels/
ignore_models_id split from SIENTIAPDE-2011/2012, not yet confirmed
with the team beyond what's in the ticket):
- Query-time filter: union of every active group's model_ids, applied
only when ALL active groups set one -- mirrors levels' exact
union-or-nothing logic in _compute_query_time_filter. Combines
cleanly with ignore_models_id's intersection $nin on the same
model_id field.
- Per-group filter: extended _check_levels_and_ignore_models_id (now
_check_levels_and_model_filters) in slot_manager.py, and the inline
condition in formatters.py's filter_notification_reports.
- Empty/missing model_ids means no restriction, same convention as
levels and ignore_models_id.
Covered by new unit tests (slot_manager, formatters,
_compute_query_time_filter) and two new e2e tests against real
Mongo/Redis/Temporal: per-group allow-list interacting with another
unrestricted group, and the query-time union excluding a model outside
every active group's allow-list.
Not yet verified on the real cluster -- blocked on the dev cluster's
package index not auto-syncing new sientia_do releases (separate issue,
being investigated for SIENTIAPDE-2013).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The library's release pipeline bumped a patch version (1.12.3), not the
minor bump (1.13.0) the feat: commit prefixes would normally produce --
correcting the pin here to match what was actually published.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Vitor asked whether limit=0 really means "no limit" on the Mongo side
and whether that had actually been tested. It's pymongo's own
documented convention, but nothing in the suite exercised it against a
real MongoDB with enough documents to mean anything -- the existing
tests only ever seed 2-5. Seeds 12 (more than any other test in this
file) with the batch size left at its config default (0) and asserts
all 12 come back in one run, against real testcontainers Mongo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removed scenarios documented with no backing test (1.2.1, 1.3.2, 1.4.1,
A.1.3) and corrected three descriptions that claimed assertions the
tests don't actually make (1.1.1's "Redis slots written", 1.3.1's
"timestamps inserted", 1.5.1's "orphan schedules removed") to match
what's really verified. A.2.1 stays as-is here -- it's already accurate
for this branch's real per-group filtering test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Also fixes the test_activities fixture, which built mongodb_config by
hand and was missing the new required notification_queue_batch_size key
-- every e2e test would have hit a KeyError without this.
New test seeds 5 notifications with a batch size of 2 and confirms three
successive runs return them oldest-first in order (n1,n2 / n3,n4 / n5),
proving the cursor only advances to what was actually returned rather
than skipping documents a capped query didn't fetch. Verified against
real testcontainers Mongo/Redis/Temporal, not mocks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A cold start or notification storm had no backstop: load_latest_data
fetched the entire matching backlog in one query. Confirmed live during
SIENTIAPDE-2012's cluster verification -- an empty Redis cursor plus a
real recurring error let a single run's query exceed Temporal's 4MB
message limit and get stuck retrying forever.
Threads a new notification_queue_batch_size config (env var
NOTIFICATION_QUEUE_BATCH_SIZE, default 5000) through to the repository's
find(), hard cap per run, no drain loop -- the incremental
last_data_timestamp cursor picks up any remainder next run, per the
parent ticket's design decision. Sorts by timestamp ascending so a
capped run always returns the oldest pending documents first; without
that, a capped batch has no guarantee of hitting the oldest documents,
and the cursor would advance past whatever it skipped.
Requires sientia_do>=1.13.0 (adds limit/sort params to
MongoDBRepository.find in both the sync and async client) -- bumped the
pin here per the parent ticket's decision to do it in the same PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
filter_notification_alerts tripped Sonar's cognitive-complexity gate
(16 > 15 allowed) from its nested loop/if structure. Split into
_filter_group_notifications (per-group notification selection) and
_determine_alert_type (Redis TTL classification), leaving the public
method as a simple loop over receiver groups. Behavior and call order
are unchanged -- verified against the existing filter_notification_alerts
unit tests by inspection (sientia_do isn't installable in this local
environment, so the real suite can't run here; CI will confirm).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The parameter-level annotation on _check_keys_and_notif_id alone
wasn't enough -- mypy needs the type at the empty list's declaration
site inside filter_notification_alerts. Also reformatted the helper's
now-longer signature to satisfy ruff.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes CI formatting failure introduced by the manual Sonar cognitive-
complexity fix (extracting the ignore-list/dedup check into its own
helper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes CI formatting failure introduced by the manual Sonar cognitive-
complexity fix (extracting the levels/ignore_models_id check into its
own helper).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends filter_notification_alerts (slot_manager.py) and
filter_notification_reports (formatters.py) to check
receiver_group.get('levels') and receiver_group.get('ignore_models_id'),
mirroring the existing contents/ignore pattern exactly. Closes the gap
SIENTIAPDE-2011 deliberately left open: the query-time union/intersection
filter widens what gets fetched for the union of all active groups, but until
now nothing narrowed each group back down to its own configured slice.
Both checks short-circuit via `not levels_list or ...` / `not
ignore_models_id_list or ...` rather than accessing notification['level']/
['model_id'] directly, so groups that don't configure either field (today's
common case, and every existing unit test fixture) never evaluate the
right-hand side and can't raise KeyError on notifications missing those keys.
ignore_models_id reads via `.get(...) or []`, defensively handling an
explicit `null` in a receiver_groups document from the start (same fix
SIENTIAPDE-2011 needed a follow-up commit for).
Tests:
- 8 new unit tests (4 per file): levels restriction, ignore_models_id
restriction, neither-configured passthrough, explicit-None safety.
- New e2e test (test_scenario_a_2_1_per_group_levels_and_ignore_models_id)
against real Mongo/Redis/Temporal: two active groups configured
differently: SIENTIAPDE-2011's query-time filter widens the fetch to the
union of both, and this subtask's per-group filter narrows each group back
to its own levels/ignore_models_id, including the case where a model is
ignored by only one of the active groups.
- Deduplicated the two e2e suites' identical _extract_html_body helper into
e2e/helpers.py (extract_html_body), now shared by both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Manual code-review pass (code-review-python checklist) flagged that no test
exercised default_filter already restricting level (e.g. Alerts' ERROR-only
default) combined with every active group specifying its own levels. The
replace-not-AND behavior is deliberate per SIENTIAPDE-2011's design doc and
ticket text, but was previously only covered by a docstring comment. Adds
test_compute_query_time_filter_union_replaces_default_level_restriction so a
future change reintroducing AND semantics fails a test instead of passing
silently.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Code review findings on the SIENTIAPDE-2011 PR:
- [Major] _compute_query_time_filter used group.get('ignore_models_id', [])
for the intersection, whose default only applies when the key is missing.
receiver_groups has no schema validation, so a document with
ignore_models_id explicitly set to null (key present, value None) made
.get() return None, and set(None) raised TypeError, crashing the whole
activity for every mail type. Fixed with `group.get(...) or []`. Added a
regression test.
- [Minor] CLAUDE.md:48 described Alerts' base data filter as always
`level: ERROR`, which this ticket's own query-time union can now widen.
Updated the line to reflect that.
- [Blocker, not applied] A finding argued the levels union should AND with
default_filter's own level restriction instead of replacing it, to avoid
widening Alerts (ERROR-only real-time channel) when a receiver_group sets
broader levels. Both SIENTIAPDE-2011's ticket text and the parent design
doc explicitly call for the union to replace the hard-coded ERROR-only
default ("whatever levels the active groups actually care about"), so
applying that fix would revert the subtask's actual purpose. The
interim gap the finding correctly identifies (no per-group level filter
exists yet until SIENTIAPDE-2012 lands) is now documented in
_compute_query_time_filter's docstring instead of "fixed" by narrowing
query-time behavior the ticket asked for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
notification['timestamp'] crosses at least one Temporal activity
boundary (load_latest_data -> filter_notification_reports/alerts)
before reaching format_log_report. Temporal's default data converter
serializes datetime values to plain ISO-8601 strings without
restoring them on decode, and pymongo returns naive datetimes for
BSON Date fields, so the string can carry no UTC offset at all. That
broke export_data_to_postgres's strict to_datetime(..., format=
DATETIME_FORMAT_WITH_TZ) parsing downstream with:
ValueError: time data "2024-06-01T12:00:00" doesn't match format
"%Y-%m-%d %H:%M:%S%z"
Same underlying issue as SIENTIAPDE-2007/2009, fixed here using the
same pattern as SlotManager.put_last_data_timestamp: normalize with
pandas.to_datetime(..., utc=True) and re-format to
DATETIME_FORMAT_WITH_TZ before the value crosses the next boundary.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI mypy caught what my local environment couldn't (Python 3.14 has no
compatible numpy stub for the project's python_version=3.11 mypy target,
so mypy couldn't run locally at all):
slot_manager.py:353: error: Incompatible types in assignment
(expression has type "str", variable has type "Timestamp")
slot_manager.py:368: error: Incompatible return value type
(got "Timestamp", expected "str | None")
Reassigning `last_data_timestamp` from a Timestamp to its strftime() string
trips mypy's inference (it fixes an unannotated local's type from its first
assignment). Split into a separate `last_data_timestamp_str` for the string
form instead of reusing the name.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
e2e fixtures (default_notification, seed_last_timestamp, default_receiver_group)
still assumed the pre-SIENTIAPDE-2007 string-timestamp representation and the
old ignore_models field name, so the load_notification_package e2e suite was
failing independently of this ticket's changes:
- default_notification now seeds `timestamp` as a native datetime (matches
production since SIENTIAPDE-2007's Notification.timestamp = now()).
- seed_last_timestamp now uses DATETIME_FORMAT_MS_WITH_TZ (matches what
put_last_data_timestamp actually writes to Redis today).
- default_receiver_group's ignore_models renamed to ignore_models_id, per
this ticket's decision.
- test_reports_main_workflow.py's explicit timestamp= call sites updated to
match.
Also fixes a real production bug in put_last_data_timestamp (slot_manager.py)
surfaced by running the e2e suite against a real Temporal environment: Temporal's
default data converter serializes datetime values to ISO-8601 strings across
the activity boundary, and pymongo returns naive datetimes for BSON Date
fields, so `data['timestamp'].max().strftime(...)` crashed on a plain str.
Unit tests never caught this because they mock the activity boundary and
never cross real (de)serialization. Fixed by normalizing through
`pandas.to_datetime(..., utc=True)`, which handles native datetime/Timestamp
and offset-aware/naive ISO strings uniformly.
Adds test_load_package_query_time_filter_union_and_intersection, a new e2e
test exercising the actual SIENTIAPDE-2011 behavior end-to-end: union of
active groups' levels, intersection of their ignore_models_id, and the
requirement that a model ignored by only one of several active groups must
not be excluded from the fetch.
Note: test_reports_main_workflow.py / test_alerts_main_workflow.py still have
a separate, pre-existing failure in process_notifications.py (same family of
bug: a notification's timestamp field parsed with a fixed strptime format
after crossing the same Temporal serialization boundary) — confirmed present
on the unmodified baseline too, unrelated to this ticket, tracked separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Push level and model-exclusion filtering down to the Mongo query instead of
fetching the full notification_queue unfiltered on every run.
- Fix await ordering in load_notification_package so sending_configs (active
receiver_groups) resolves before load_latest_data is called, letting the
query filter be built from group data.
- Add _compute_query_time_filter: unions active groups' `levels` (falls back
to the workflow's default filter if any group has no restriction, since
that group wants every level) and intersects their `ignore_models_id`
(a model is only safe to drop from the fetch if every active group
ignores it).
- Add unit tests covering mixed groups, a model ignored by only one group
(must not be excluded from the fetch), and the empty active-groups case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses review feedback: prefer sientia_do's explicit datetime format
constant over Python's isoformat()/fromisoformat() when round-tripping
last_data_timestamp through Redis, matching the convention already used
elsewhere in mongo_db.py and slot_manager.py.
Notification.timestamp is now stored as a native BSON Date instead of a
string, so load_latest_data must parse the ISO string coming from Redis
back to datetime before building the $gt filter, or the comparison
silently breaks (BSON String vs Date always evaluates False). Also fixes
put_last_data_timestamp, which would otherwise raise TypeError trying to
json.dumps a pandas Timestamp when persisting the new watermark to Redis.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
