SIENTIAPDE-1974-connectors-pi-web-api: Introduce config_hash for robust schedule redeployment. Pipelines now redeploy on any content change (e.g., tag database updates) even without an updated_at bump. orchestrated_schedules tracks the hash; updated documentation, E2E tests, and the SIENTIAPDE-1974 backfill script.
SIENTIAPDE-1974 Write OPC tags through the tag database and drop the PI Web API path
- predictions_batch write_tags are now [{tag_id}] only: resolve_write_tags builds
the opc_output_config from connector_tags + connectors (server = connector_id,
Write / Prediction Confidence -> prediction_tags / confidence_tags, address and
data_type from the tag, float by default). The legacy server_id/type/addr write
tags are no longer read and are notified as invalid.
- A pipeline's effective updated_at is the latest of its own and of the
connector_tags it references, so renaming, moving or deactivating a tag
redeploys it (removes the rename limitation).
- The connector_tags query loads the referenced tags of every category and
status; the code filters them. aggregate_documents_in_mongodb formats only
datetime timestamp fields.
- Remove the PI Web API tag database path (connectors-pi-web-api patch, PI loads,
resolution, connection and held schedules): PI pipelines behave as on main.
- tag_id is canonical text and the workflow-side id collection never raises;
a malformed pipeline no longer fails the whole OPC slot.
- Replay test against histories recorded on the e2e stack (main and this
version), e2e scenarios for write tags and tag changes, and the runbook
script scripts/sientiapde_1974_backfill_write_tags.js.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-1974 Split orchestrator_functions into pipeline configs, tag database and OPC slots
- orchestrator_functions.py held four unrelated concerns (pipeline
builders, tag database indexes, PI Web API resolution, OPC slot v2)
and had grown to 1135 lines. Move them, unchanged, into:
pipeline_configs.py, tag_database.py and opc_slots.py.
- Make the helpers shared across modules public: is_complete,
tag_id_text and pi_connection.
- Split the utils tests the same way, with the pipeline and tag
database documents they share in tag_database_fixtures.py.
- Point formatters, the orchestrator workflow, README and CLAUDE.md at
the new modules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add e2e tag-database scenarios and cover worker/email/formatter gaps
- Add e2e scenarios 1.9.1-1.9.4 for PI Web API through the tag
database: scouter and output tag resolution, mixed-connector schedule
hold, and OPC slot unchanged by PI connectors. Seed connectors and
connector_tags from a new scenario input.
- Tighten the happy-path e2e to assert both schedules and that no PI
output config is produced when the tag database is unused.
- Drop dead `if self.server` / `if notification_handler` guards in the
email reconnect and worker shutdown paths, since both are always set.
- Log the traceback inside the message on worker failure, as the
sientia_do Logger.error does not accept exc_info.
- Add unit tests for the worker failure exit, Prometheus startup
failure, email reconnect without a password, and formatter, mongo,
temporal manager and orchestrator edge cases.
SIENTIAPDE-1974 Extract PI tag resolution and name Mongo stage constants
- Move PI Web API read/output tag resolution out of the schedule loop
into _resolve_pi_pipeline, so the loop only handles the held case.
- Replace repeated '$match'/'$project' literals in the default
connector queries with MATCH/PROJECT constants.
Resolve PI Web API pipeline tags through the tag database
- Load active `pi_web_api` connectors and referenced tags behind the
`connectors-pi-web-api` patch, so replays of older runs are unaffected.
- Resolve `pi_web_api_scouter` read tags and `predictions_batch` output
tags by `tag_id`; legacy `webid` pipelines produce identical input.
- Add a `connection` block to scouter schedules with the secret reference
only, plus connector defaults for period, max_count and api_timeout.
- Hold pipelines whose tags span several connectors and notify unresolved
tags, wrong categories and missing connectors once per run.
- Share the issue notification logic between slots and schedules.
SIENTIAPDE-1974: merge main (0.8.2) into the tag database feature
Brings release/SIENTIAPDE-1974 (slot v2) up to date with main (R1).
Conflicts: both sides kept in orchestrator_functions.py imports, test
imports and README; version follows main (bump is automated on merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-2212-drift-janela-e-chunk-period: Remove default 1-minute margin from drift window calculation. The default interval now directly matches the frequency, ensuring consecutive windows do not overlap.
Add e2e scenario for drift window and chunk_period
- Add scenario 1.8.1: two drift pipelines on frequency 1m. One has no
interval_minutes and an invalid chunk_period='hour', so it expects
interval=2, chunk_period='min' and one DRIFT_CHUNK_PERIOD_INVALID
notification. The other sets interval_minutes=60 and chunk_period='s',
and both are forwarded unchanged with no warning (SIENTIAPDE-2212).
- Add a _schedule_input helper that decodes the workflow input of a
laborious-namespace schedule.
- Reformat the ensure_namespace_search_attributes signature in
conftest.py to a single line.
SIENTIAPDE-2212-drift-janela-e-chunk-period: Add configurable chunk period and dynamic window calculation for drift, with warnings for invalid periods.
reusable-deploy-dataops.yml encadeia release-tarball -> dataops-module-release.yml,
que pede pull-requests:read; GitHub valida a interseção de permissions em todo o
workflow_call encadeado, independente do if: que gate o job. Removido por engano
em 586a5b9, achando que só existia por causa do trigger pull_request antigo.
[skip ci]
Caller workflow had no permissions block, so GitHub's permission-
intersection rule capped the nested reusable-version-bump.yml call
to read-only, breaking dispatch ("requesting 'contents: write', but
is only allowed 'contents: read'").
[skip ci]
New workflow bumps pyproject.toml's version on PR merge, calling the
shared reusable-version-bump.yml (SIENTIAPDE-2217 version-bump reform) —
zero build/image/chart/deploy side effects. deploy.yml (workflow_dispatch-
only) will read this committed value instead of calculating one.
[skip ci]
Removes auto-deploy-on-PR-merge trigger and the old PR-merge gate. Deploy now
triggers only via workflow_dispatch, with chart_ref required (no default) as
the literal sientia-helm-chart branch bumped and deployed — hard error if it
doesn't exist, no silent fallback to development/main.
- .github/workflows/deploy.yml: drop pull_request trigger, add chart_ref
input, branch_name now github.ref_name, drop pull-requests:read permission
- Delete .github/workflows/require-chart-ref-label.yml
[skip ci]
sientia_do 1.12.5 pins the psycopg2 driver in the Postgres engine URL.
Without it, rebuilding the image pulls SQLAlchemy 2.1.x, whose default
'postgresql://' driver is psycopg (v3), and the worker crashes at
startup with "ModuleNotFoundError: No module named 'psycopg'".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SonarQube on PR #45 reported 3 new issues; all fixed without behavior change:
- S3776 gather_read_tags (cognitive complexity 22): the merge of one resolved
read tag into its (server, address) entry moves to _add_read_tag (now 14).
- S3776 Orchestrator.run (16): the tag database load moves to
Orchestrator._load_tag_database (run back to 14). workflow.patched() is still
evaluated first on every execution, so its marker is always recorded.
- S1192 '$match' duplicated in the workflow module: the connector_tags tag_id
filter moves to orchestrator_functions.filter_by_tag_ids, which also returns a
new aggregation list.
194 unit tests (new: filter_by_tag_ids); real workflow on a local Temporal test
server still passes the 3 scenarios; the dev dry run is still equivalent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review (code-review skill, xhigh) of efa0d94: 8 findings, 6 fixed here,
2 documented (decisions for the team).
Fixed:
- The tag database is read only when a scouter read tag carries a tag_id, and
only the referenced tags ($match tag_id $in, int and text forms), through
aggregations that project only the fields the slot uses. Environments without
a tag database, or whose pipelines do not use tag_id, never query it (P4), and
other attributes (e.g. an ObjectId, which the Temporal JSON converter rejects)
never reach a payload. Fresh aggregation lists are passed, because
aggregate_documents_in_mongodb appends a stage to the list it gets.
- Incomplete connectors / connector_tags documents are left out of the indexes
(their read tags are notified as not found) instead of raising KeyError and
failing the whole run.
- frequency is stored as int, so 15000 vs '15000' across pipelines no longer
makes the slot depend on the pipeline order (O6).
- A missing, null or empty tag_id means "no tag database": an empty string no
longer drops a legacy read tag.
- A read tag without an integer frequency (or, without tag_id, without
server_id / tag_address / tag_name) is skipped and notified
(ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION) instead of failing
the slot build.
- Server slot keys can no longer collide (name:server_id is extended until free).
Documented in the README:
- The gateway's plan B (READ_ENGINE=legacy) reads v1 only: switching to it after
this deploy also requires rolling the Orchestrator back.
- Renaming a tag in connector_tags does not redeploy the scouter (schedules
update on the pipeline's updated_at).
Validation: 193 unit tests; the real workflow on a local Temporal test server
(sandbox on) with the real Formatters activities and the dev data: slot v2
written without querying the tag database, a second run inserts nothing, and a
tag_id pipeline queries both aggregations with the $in filter; the dev dry run
is still equivalent to the current v1 slot through the gateway's parse_slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Orchestrator now writes the slot v2 (contract C1) the OPC Gateway reads:
one entry per (server, address) with a names[] list, so an address read by N
pipelines is read once and written to N collections (fixes "one name per
address", where the second model's scouter discarded everything).
- O1: load connectors (opc_ua, Active) and connector_tags (Read, Active) in
parallel with the current loads. Queries are optional in the input with
defaults in code (D9); empty or missing collections load nothing. The loads
sit behind workflow.patched('SIENTIAPDE-1974-tag-database'), so a run in
flight during a deploy replays without them.
- O2: gather_read_tags groups by (server, address). A read tag with tag_id
resolves address/connector from connector_tags and the server from connectors
(RF3); without tag_id it keeps server_id + tag_address + opc-servers (RF4),
so every environment without a tag database keeps collecting (P4).
- O3/SC1: resolve_tag_names is the single source of tag names for both the
scouter's model_tags and the slot's names[].
- O4: build_tag_config emits server_id/name/url/server_uri/use_security/
subscription_period_ms + tags{frequency, active_read, names}; no certificate
paths (the gateway takes the certificate from MinIO). frequency = lowest
among pipelines; subscription period = connector value, else min/2 as before;
active_read off unless a pipeline enables it (lowest cycle_multiple, D3).
- O5: process_slots splits by address (names of an address never split) and
notifies, once per run, unknown tag_id, missing/inactive connector and
unknown opc-servers server_id.
- O6: create_slot_config writes only new or changed slots; the slot is built
in a stable order, so unchanged pipelines do not make the gateway
re-subscribe (and the "Inserted slots" report stops firing every run).
- O7: unit tests for multi-name, repeated address, unknown tag_id, inactive
connector, mixed tags, server-name clash, active_read, slot division,
stability, SC1 and the replay path (184 tests).
- O8: README documents the slot v2 contract and the new optional inputs.
- test_email.test_close: count only the instance under test; an Email left by
another test can be garbage-collected (Email.__del__ -> close) while the
patch is active, which the new tests' allocation pattern exposed.
Dry run with the dev data (scouter-sin-approx): the v2 slot is equivalent to
the current v1 slot as seen by the gateway's parse_slot (same server, addresses,
collections, names, frequency and subscription period).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the review-feedback fixes (batch-size default, requirements-local,
docstring trim) from PR #43 so #44 stays consistent with its base branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Conflicts:
# CLAUDE.md
- NOTIFICATION_QUEUE_BATCH_SIZE default reverted to 0 (unlimited) -- the
existing behavior before this ticket, so deployments must opt in to the
cap rather than having it silently applied at 5000
- requirements-local.txt: bump sientia_do pin to 1.12.3 to match requirements.txt
- Trim the load_latest_data docstring addition so it reads as documentation,
not a changelog entry
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
