SIENTIAPDE-1974 Extract PI tag resolution and name Mongo stage constants
- Move PI Web API read/output tag resolution out of the schedule loop
into _resolve_pi_pipeline, so the loop only handles the held case.
- Replace repeated '$match'/'$project' literals in the default
connector queries with MATCH/PROJECT constants.
Resolve PI Web API pipeline tags through the tag database
- Load active `pi_web_api` connectors and referenced tags behind the
`connectors-pi-web-api` patch, so replays of older runs are unaffected.
- Resolve `pi_web_api_scouter` read tags and `predictions_batch` output
tags by `tag_id`; legacy `webid` pipelines produce identical input.
- Add a `connection` block to scouter schedules with the secret reference
only, plus connector defaults for period, max_count and api_timeout.
- Hold pipelines whose tags span several connectors and notify unresolved
tags, wrong categories and missing connectors once per run.
- Share the issue notification logic between slots and schedules.
SIENTIAPDE-1974: merge main (0.8.2) into the tag database feature
Brings release/SIENTIAPDE-1974 (slot v2) up to date with main (R1).
Conflicts: both sides kept in orchestrator_functions.py imports, test
imports and README; version follows main (bump is automated on merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-2212-drift-janela-e-chunk-period: Remove default 1-minute margin from drift window calculation. The default interval now directly matches the frequency, ensuring consecutive windows do not overlap.
Add e2e scenario for drift window and chunk_period
- Add scenario 1.8.1: two drift pipelines on frequency 1m. One has no
interval_minutes and an invalid chunk_period='hour', so it expects
interval=2, chunk_period='min' and one DRIFT_CHUNK_PERIOD_INVALID
notification. The other sets interval_minutes=60 and chunk_period='s',
and both are forwarded unchanged with no warning (SIENTIAPDE-2212).
- Add a _schedule_input helper that decodes the workflow input of a
laborious-namespace schedule.
- Reformat the ensure_namespace_search_attributes signature in
conftest.py to a single line.
SIENTIAPDE-2212-drift-janela-e-chunk-period: Add configurable chunk period and dynamic window calculation for drift, with warnings for invalid periods.
reusable-deploy-dataops.yml encadeia release-tarball -> dataops-module-release.yml,
que pede pull-requests:read; GitHub valida a interseção de permissions em todo o
workflow_call encadeado, independente do if: que gate o job. Removido por engano
em 586a5b9, achando que só existia por causa do trigger pull_request antigo.
[skip ci]
Caller workflow had no permissions block, so GitHub's permission-
intersection rule capped the nested reusable-version-bump.yml call
to read-only, breaking dispatch ("requesting 'contents: write', but
is only allowed 'contents: read'").
[skip ci]
New workflow bumps pyproject.toml's version on PR merge, calling the
shared reusable-version-bump.yml (SIENTIAPDE-2217 version-bump reform) —
zero build/image/chart/deploy side effects. deploy.yml (workflow_dispatch-
only) will read this committed value instead of calculating one.
[skip ci]
Removes auto-deploy-on-PR-merge trigger and the old PR-merge gate. Deploy now
triggers only via workflow_dispatch, with chart_ref required (no default) as
the literal sientia-helm-chart branch bumped and deployed — hard error if it
doesn't exist, no silent fallback to development/main.
- .github/workflows/deploy.yml: drop pull_request trigger, add chart_ref
input, branch_name now github.ref_name, drop pull-requests:read permission
- Delete .github/workflows/require-chart-ref-label.yml
[skip ci]
sientia_do 1.12.5 pins the psycopg2 driver in the Postgres engine URL.
Without it, rebuilding the image pulls SQLAlchemy 2.1.x, whose default
'postgresql://' driver is psycopg (v3), and the worker crashes at
startup with "ModuleNotFoundError: No module named 'psycopg'".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SonarQube on PR #45 reported 3 new issues; all fixed without behavior change:
- S3776 gather_read_tags (cognitive complexity 22): the merge of one resolved
read tag into its (server, address) entry moves to _add_read_tag (now 14).
- S3776 Orchestrator.run (16): the tag database load moves to
Orchestrator._load_tag_database (run back to 14). workflow.patched() is still
evaluated first on every execution, so its marker is always recorded.
- S1192 '$match' duplicated in the workflow module: the connector_tags tag_id
filter moves to orchestrator_functions.filter_by_tag_ids, which also returns a
new aggregation list.
194 unit tests (new: filter_by_tag_ids); real workflow on a local Temporal test
server still passes the 3 scenarios; the dev dry run is still equivalent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review (code-review skill, xhigh) of efa0d94: 8 findings, 6 fixed here,
2 documented (decisions for the team).
Fixed:
- The tag database is read only when a scouter read tag carries a tag_id, and
only the referenced tags ($match tag_id $in, int and text forms), through
aggregations that project only the fields the slot uses. Environments without
a tag database, or whose pipelines do not use tag_id, never query it (P4), and
other attributes (e.g. an ObjectId, which the Temporal JSON converter rejects)
never reach a payload. Fresh aggregation lists are passed, because
aggregate_documents_in_mongodb appends a stage to the list it gets.
- Incomplete connectors / connector_tags documents are left out of the indexes
(their read tags are notified as not found) instead of raising KeyError and
failing the whole run.
- frequency is stored as int, so 15000 vs '15000' across pipelines no longer
makes the slot depend on the pipeline order (O6).
- A missing, null or empty tag_id means "no tag database": an empty string no
longer drops a legacy read tag.
- A read tag without an integer frequency (or, without tag_id, without
server_id / tag_address / tag_name) is skipped and notified
(ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION) instead of failing
the slot build.
- Server slot keys can no longer collide (name:server_id is extended until free).
Documented in the README:
- The gateway's plan B (READ_ENGINE=legacy) reads v1 only: switching to it after
this deploy also requires rolling the Orchestrator back.
- Renaming a tag in connector_tags does not redeploy the scouter (schedules
update on the pipeline's updated_at).
Validation: 193 unit tests; the real workflow on a local Temporal test server
(sandbox on) with the real Formatters activities and the dev data: slot v2
written without querying the tag database, a second run inserts nothing, and a
tag_id pipeline queries both aggregations with the $in filter; the dev dry run
is still equivalent to the current v1 slot through the gateway's parse_slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Orchestrator now writes the slot v2 (contract C1) the OPC Gateway reads:
one entry per (server, address) with a names[] list, so an address read by N
pipelines is read once and written to N collections (fixes "one name per
address", where the second model's scouter discarded everything).
- O1: load connectors (opc_ua, Active) and connector_tags (Read, Active) in
parallel with the current loads. Queries are optional in the input with
defaults in code (D9); empty or missing collections load nothing. The loads
sit behind workflow.patched('SIENTIAPDE-1974-tag-database'), so a run in
flight during a deploy replays without them.
- O2: gather_read_tags groups by (server, address). A read tag with tag_id
resolves address/connector from connector_tags and the server from connectors
(RF3); without tag_id it keeps server_id + tag_address + opc-servers (RF4),
so every environment without a tag database keeps collecting (P4).
- O3/SC1: resolve_tag_names is the single source of tag names for both the
scouter's model_tags and the slot's names[].
- O4: build_tag_config emits server_id/name/url/server_uri/use_security/
subscription_period_ms + tags{frequency, active_read, names}; no certificate
paths (the gateway takes the certificate from MinIO). frequency = lowest
among pipelines; subscription period = connector value, else min/2 as before;
active_read off unless a pipeline enables it (lowest cycle_multiple, D3).
- O5: process_slots splits by address (names of an address never split) and
notifies, once per run, unknown tag_id, missing/inactive connector and
unknown opc-servers server_id.
- O6: create_slot_config writes only new or changed slots; the slot is built
in a stable order, so unchanged pipelines do not make the gateway
re-subscribe (and the "Inserted slots" report stops firing every run).
- O7: unit tests for multi-name, repeated address, unknown tag_id, inactive
connector, mixed tags, server-name clash, active_read, slot division,
stability, SC1 and the replay path (184 tests).
- O8: README documents the slot v2 contract and the new optional inputs.
- test_email.test_close: count only the instance under test; an Email left by
another test can be garbage-collected (Email.__del__ -> close) while the
patch is active, which the new tests' allocation pattern exposed.
Dry run with the dev data (scouter-sin-approx): the v2 slot is equivalent to
the current v1 slot as seen by the gateway's parse_slot (same server, addresses,
collections, names, frequency and subscription period).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the review-feedback fixes (batch-size default, requirements-local,
docstring trim) from PR #43 so #44 stays consistent with its base branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts:
# CLAUDE.md
- NOTIFICATION_QUEUE_BATCH_SIZE default reverted to 0 (unlimited) -- the
existing behavior before this ticket, so deployments must opt in to the
cap rather than having it silently applied at 5000
- requirements-local.txt: bump sientia_do pin to 1.12.3 to match requirements.txt
- Trim the load_latest_data docstring addition so it reads as documentation,
not a changelog entry
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
levels and model_ids (SIENTIAPDE-2149) both computed a per-group union with
identical "union only if every active group restricts, else unrestricted"
logic. Extracting _union_when_all_restrict removes the duplicated for/if
block for each axis, addressing the SonarQube quality gate failure on this
PR (cognitive complexity).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New, separate model_id-based filter distinct from the existing
ignore_models_id deny-list (SIENTIAPDE-2012): a group configuring
model_ids only receives notifications from those models. Both axes
apply together (AND) when a group sets both.
Design decisions made while implementing (matching the levels/
ignore_models_id split from SIENTIAPDE-2011/2012, not yet confirmed
with the team beyond what's in the ticket):
- Query-time filter: union of every active group's model_ids, applied
only when ALL active groups set one -- mirrors levels' exact
union-or-nothing logic in _compute_query_time_filter. Combines
cleanly with ignore_models_id's intersection $nin on the same
model_id field.
- Per-group filter: extended _check_levels_and_ignore_models_id (now
_check_levels_and_model_filters) in slot_manager.py, and the inline
condition in formatters.py's filter_notification_reports.
- Empty/missing model_ids means no restriction, same convention as
levels and ignore_models_id.
Covered by new unit tests (slot_manager, formatters,
_compute_query_time_filter) and two new e2e tests against real
Mongo/Redis/Temporal: per-group allow-list interacting with another
unrestricted group, and the query-time union excluding a model outside
every active group's allow-list.
Not yet verified on the real cluster -- blocked on the dev cluster's
package index not auto-syncing new sientia_do releases (separate issue,
being investigated for SIENTIAPDE-2013).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The library's release pipeline bumped a patch version (1.12.3), not the
minor bump (1.13.0) the feat: commit prefixes would normally produce --
correcting the pin here to match what was actually published.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
