SIENTIAPDE-1974-connectors-pi-web-api: Consolidate OPC server and tag data sourcing to the tag database. This change removes the legacy opc-servers collection and enforces tag_id as the sole reference for all read and write tags via connectors and connector_tags. The Orchestrator workflow now exclusively loads OPC server and tag information from the tag database, simplifying data flow and removing conditional logic for legacy data sources. Improved error reporting for pipeline timestamp updates and updated documentation, examples, and tests are also included.
SIENTIAPDE-1974-connectors-pi-web-api: Remove write tag backfill script and update README to require pre-deployment conversion of write tags to [{tag_id}] format.
SIENTIAPDE-1974-connectors-pi-web-api: Introduce config_hash for robust schedule redeployment. Pipelines now redeploy on any content change (e.g., tag database updates) even without an updated_at bump. orchestrated_schedules tracks the hash; updated documentation, E2E tests, and the SIENTIAPDE-1974 backfill script.
SIENTIAPDE-1974 Write OPC tags through the tag database and drop the PI Web API path
- predictions_batch write_tags are now [{tag_id}] only: resolve_write_tags builds
the opc_output_config from connector_tags + connectors (server = connector_id,
Write / Prediction Confidence -> prediction_tags / confidence_tags, address and
data_type from the tag, float by default). The legacy server_id/type/addr write
tags are no longer read and are notified as invalid.
- A pipeline's effective updated_at is the latest of its own and of the
connector_tags it references, so renaming, moving or deactivating a tag
redeploys it (removes the rename limitation).
- The connector_tags query loads the referenced tags of every category and
status; the code filters them. aggregate_documents_in_mongodb formats only
datetime timestamp fields.
- Remove the PI Web API tag database path (connectors-pi-web-api patch, PI loads,
resolution, connection and held schedules): PI pipelines behave as on main.
- tag_id is canonical text and the workflow-side id collection never raises;
a malformed pipeline no longer fails the whole OPC slot.
- Replay test against histories recorded on the e2e stack (main and this
version), e2e scenarios for write tags and tag changes, and the runbook
script scripts/sientiapde_1974_backfill_write_tags.js.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-1974 Split orchestrator_functions into pipeline configs, tag database and OPC slots
- orchestrator_functions.py held four unrelated concerns (pipeline
builders, tag database indexes, PI Web API resolution, OPC slot v2)
and had grown to 1135 lines. Move them, unchanged, into:
pipeline_configs.py, tag_database.py and opc_slots.py.
- Make the helpers shared across modules public: is_complete,
tag_id_text and pi_connection.
- Split the utils tests the same way, with the pipeline and tag
database documents they share in tag_database_fixtures.py.
- Point formatters, the orchestrator workflow, README and CLAUDE.md at
the new modules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add e2e tag-database scenarios and cover worker/email/formatter gaps
- Add e2e scenarios 1.9.1-1.9.4 for PI Web API through the tag
database: scouter and output tag resolution, mixed-connector schedule
hold, and OPC slot unchanged by PI connectors. Seed connectors and
connector_tags from a new scenario input.
- Tighten the happy-path e2e to assert both schedules and that no PI
output config is produced when the tag database is unused.
- Drop dead `if self.server` / `if notification_handler` guards in the
email reconnect and worker shutdown paths, since both are always set.
- Log the traceback inside the message on worker failure, as the
sientia_do Logger.error does not accept exc_info.
- Add unit tests for the worker failure exit, Prometheus startup
failure, email reconnect without a password, and formatter, mongo,
temporal manager and orchestrator edge cases.
SIENTIAPDE-1974 Extract PI tag resolution and name Mongo stage constants
- Move PI Web API read/output tag resolution out of the schedule loop
into _resolve_pi_pipeline, so the loop only handles the held case.
- Replace repeated '$match'/'$project' literals in the default
connector queries with MATCH/PROJECT constants.
Resolve PI Web API pipeline tags through the tag database
- Load active `pi_web_api` connectors and referenced tags behind the
`connectors-pi-web-api` patch, so replays of older runs are unaffected.
- Resolve `pi_web_api_scouter` read tags and `predictions_batch` output
tags by `tag_id`; legacy `webid` pipelines produce identical input.
- Add a `connection` block to scouter schedules with the secret reference
only, plus connector defaults for period, max_count and api_timeout.
- Hold pipelines whose tags span several connectors and notify unresolved
tags, wrong categories and missing connectors once per run.
- Share the issue notification logic between slots and schedules.
SIENTIAPDE-1974: merge main (0.8.2) into the tag database feature
Brings release/SIENTIAPDE-1974 (slot v2) up to date with main (R1).
Conflicts: both sides kept in orchestrator_functions.py imports, test
imports and README; version follows main (bump is automated on merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-2212-drift-janela-e-chunk-period: Remove default 1-minute margin from drift window calculation. The default interval now directly matches the frequency, ensuring consecutive windows do not overlap.
Add e2e scenario for drift window and chunk_period
- Add scenario 1.8.1: two drift pipelines on frequency 1m. One has no
interval_minutes and an invalid chunk_period='hour', so it expects
interval=2, chunk_period='min' and one DRIFT_CHUNK_PERIOD_INVALID
notification. The other sets interval_minutes=60 and chunk_period='s',
and both are forwarded unchanged with no warning (SIENTIAPDE-2212).
- Add a _schedule_input helper that decodes the workflow input of a
laborious-namespace schedule.
- Reformat the ensure_namespace_search_attributes signature in
conftest.py to a single line.
SIENTIAPDE-2212-drift-janela-e-chunk-period: Add configurable chunk period and dynamic window calculation for drift, with warnings for invalid periods.
reusable-deploy-dataops.yml encadeia release-tarball -> dataops-module-release.yml,
que pede pull-requests:read; GitHub valida a interseção de permissions em todo o
workflow_call encadeado, independente do if: que gate o job. Removido por engano
em 586a5b9, achando que só existia por causa do trigger pull_request antigo.
[skip ci]
Caller workflow had no permissions block, so GitHub's permission-
intersection rule capped the nested reusable-version-bump.yml call
to read-only, breaking dispatch ("requesting 'contents: write', but
is only allowed 'contents: read'").
[skip ci]
New workflow bumps pyproject.toml's version on PR merge, calling the
shared reusable-version-bump.yml (SIENTIAPDE-2217 version-bump reform) —
zero build/image/chart/deploy side effects. deploy.yml (workflow_dispatch-
only) will read this committed value instead of calculating one.
[skip ci]
Removes auto-deploy-on-PR-merge trigger and the old PR-merge gate. Deploy now
triggers only via workflow_dispatch, with chart_ref required (no default) as
the literal sientia-helm-chart branch bumped and deployed — hard error if it
doesn't exist, no silent fallback to development/main.
- .github/workflows/deploy.yml: drop pull_request trigger, add chart_ref
input, branch_name now github.ref_name, drop pull-requests:read permission
- Delete .github/workflows/require-chart-ref-label.yml
[skip ci]
sientia_do 1.12.5 pins the psycopg2 driver in the Postgres engine URL.
Without it, rebuilding the image pulls SQLAlchemy 2.1.x, whose default
'postgresql://' driver is psycopg (v3), and the worker crashes at
startup with "ModuleNotFoundError: No module named 'psycopg'".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SonarQube on PR #45 reported 3 new issues; all fixed without behavior change:
- S3776 gather_read_tags (cognitive complexity 22): the merge of one resolved
read tag into its (server, address) entry moves to _add_read_tag (now 14).
- S3776 Orchestrator.run (16): the tag database load moves to
Orchestrator._load_tag_database (run back to 14). workflow.patched() is still
evaluated first on every execution, so its marker is always recorded.
- S1192 '$match' duplicated in the workflow module: the connector_tags tag_id
filter moves to orchestrator_functions.filter_by_tag_ids, which also returns a
new aggregation list.
194 unit tests (new: filter_by_tag_ids); real workflow on a local Temporal test
server still passes the 3 scenarios; the dev dry run is still equivalent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review (code-review skill, xhigh) of efa0d94: 8 findings, 6 fixed here,
2 documented (decisions for the team).
Fixed:
- The tag database is read only when a scouter read tag carries a tag_id, and
only the referenced tags ($match tag_id $in, int and text forms), through
aggregations that project only the fields the slot uses. Environments without
a tag database, or whose pipelines do not use tag_id, never query it (P4), and
other attributes (e.g. an ObjectId, which the Temporal JSON converter rejects)
never reach a payload. Fresh aggregation lists are passed, because
aggregate_documents_in_mongodb appends a stage to the list it gets.
- Incomplete connectors / connector_tags documents are left out of the indexes
(their read tags are notified as not found) instead of raising KeyError and
failing the whole run.
- frequency is stored as int, so 15000 vs '15000' across pipelines no longer
makes the slot depend on the pipeline order (O6).
- A missing, null or empty tag_id means "no tag database": an empty string no
longer drops a legacy read tag.
- A read tag without an integer frequency (or, without tag_id, without
server_id / tag_address / tag_name) is skipped and notified
(ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION) instead of failing
the slot build.
- Server slot keys can no longer collide (name:server_id is extended until free).
Documented in the README:
- The gateway's plan B (READ_ENGINE=legacy) reads v1 only: switching to it after
this deploy also requires rolling the Orchestrator back.
- Renaming a tag in connector_tags does not redeploy the scouter (schedules
update on the pipeline's updated_at).
Validation: 193 unit tests; the real workflow on a local Temporal test server
(sandbox on) with the real Formatters activities and the dev data: slot v2
written without querying the tag database, a second run inserts nothing, and a
tag_id pipeline queries both aggregations with the $in filter; the dev dry run
is still equivalent to the current v1 slot through the gateway's parse_slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Orchestrator now writes the slot v2 (contract C1) the OPC Gateway reads:
one entry per (server, address) with a names[] list, so an address read by N
pipelines is read once and written to N collections (fixes "one name per
address", where the second model's scouter discarded everything).
- O1: load connectors (opc_ua, Active) and connector_tags (Read, Active) in
parallel with the current loads. Queries are optional in the input with
defaults in code (D9); empty or missing collections load nothing. The loads
sit behind workflow.patched('SIENTIAPDE-1974-tag-database'), so a run in
flight during a deploy replays without them.
- O2: gather_read_tags groups by (server, address). A read tag with tag_id
resolves address/connector from connector_tags and the server from connectors
(RF3); without tag_id it keeps server_id + tag_address + opc-servers (RF4),
so every environment without a tag database keeps collecting (P4).
- O3/SC1: resolve_tag_names is the single source of tag names for both the
scouter's model_tags and the slot's names[].
- O4: build_tag_config emits server_id/name/url/server_uri/use_security/
subscription_period_ms + tags{frequency, active_read, names}; no certificate
paths (the gateway takes the certificate from MinIO). frequency = lowest
among pipelines; subscription period = connector value, else min/2 as before;
active_read off unless a pipeline enables it (lowest cycle_multiple, D3).
- O5: process_slots splits by address (names of an address never split) and
notifies, once per run, unknown tag_id, missing/inactive connector and
unknown opc-servers server_id.
- O6: create_slot_config writes only new or changed slots; the slot is built
in a stable order, so unchanged pipelines do not make the gateway
re-subscribe (and the "Inserted slots" report stops firing every run).
- O7: unit tests for multi-name, repeated address, unknown tag_id, inactive
connector, mixed tags, server-name clash, active_read, slot division,
stability, SC1 and the replay path (184 tests).
- O8: README documents the slot v2 contract and the new optional inputs.
- test_email.test_close: count only the instance under test; an Email left by
another test can be garbage-collected (Email.__del__ -> close) while the
patch is active, which the new tests' allocation pattern exposed.
Dry run with the dev data (scouter-sin-approx): the v2 slot is equivalent to
the current v1 slot as seen by the gateway's parse_slot (same server, addresses,
collections, names, frequency and subscription period).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
