Resolve PI Web API pipeline tags through the tag database
- Load active `pi_web_api` connectors and referenced tags behind the
`connectors-pi-web-api` patch, so replays of older runs are unaffected.
- Resolve `pi_web_api_scouter` read tags and `predictions_batch` output
tags by `tag_id`; legacy `webid` pipelines produce identical input.
- Add a `connection` block to scouter schedules with the secret reference
only, plus connector defaults for period, max_count and api_timeout.
- Hold pipelines whose tags span several connectors and notify unresolved
tags, wrong categories and missing connectors once per run.
- Share the issue notification logic between slots and schedules.
SIENTIAPDE-1974: merge main (0.8.2) into the tag database feature
Brings release/SIENTIAPDE-1974 (slot v2) up to date with main (R1).
Conflicts: both sides kept in orchestrator_functions.py imports, test
imports and README; version follows main (bump is automated on merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIENTIAPDE-2212-drift-janela-e-chunk-period: Remove default 1-minute margin from drift window calculation. The default interval now directly matches the frequency, ensuring consecutive windows do not overlap.
SIENTIAPDE-2212-drift-janela-e-chunk-period: Add configurable chunk period and dynamic window calculation for drift, with warnings for invalid periods.
SonarQube on PR #45 reported 3 new issues; all fixed without behavior change:
- S3776 gather_read_tags (cognitive complexity 22): the merge of one resolved
read tag into its (server, address) entry moves to _add_read_tag (now 14).
- S3776 Orchestrator.run (16): the tag database load moves to
Orchestrator._load_tag_database (run back to 14). workflow.patched() is still
evaluated first on every execution, so its marker is always recorded.
- S1192 '$match' duplicated in the workflow module: the connector_tags tag_id
filter moves to orchestrator_functions.filter_by_tag_ids, which also returns a
new aggregation list.
194 unit tests (new: filter_by_tag_ids); real workflow on a local Temporal test
server still passes the 3 scenarios; the dev dry run is still equivalent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review (code-review skill, xhigh) of efa0d94: 8 findings, 6 fixed here,
2 documented (decisions for the team).
Fixed:
- The tag database is read only when a scouter read tag carries a tag_id, and
only the referenced tags ($match tag_id $in, int and text forms), through
aggregations that project only the fields the slot uses. Environments without
a tag database, or whose pipelines do not use tag_id, never query it (P4), and
other attributes (e.g. an ObjectId, which the Temporal JSON converter rejects)
never reach a payload. Fresh aggregation lists are passed, because
aggregate_documents_in_mongodb appends a stage to the list it gets.
- Incomplete connectors / connector_tags documents are left out of the indexes
(their read tags are notified as not found) instead of raising KeyError and
failing the whole run.
- frequency is stored as int, so 15000 vs '15000' across pipelines no longer
makes the slot depend on the pipeline order (O6).
- A missing, null or empty tag_id means "no tag database": an empty string no
longer drops a legacy read tag.
- A read tag without an integer frequency (or, without tag_id, without
server_id / tag_address / tag_name) is skipped and notified
(ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION) instead of failing
the slot build.
- Server slot keys can no longer collide (name:server_id is extended until free).
Documented in the README:
- The gateway's plan B (READ_ENGINE=legacy) reads v1 only: switching to it after
this deploy also requires rolling the Orchestrator back.
- Renaming a tag in connector_tags does not redeploy the scouter (schedules
update on the pipeline's updated_at).
Validation: 193 unit tests; the real workflow on a local Temporal test server
(sandbox on) with the real Formatters activities and the dev data: slot v2
written without querying the tag database, a second run inserts nothing, and a
tag_id pipeline queries both aggregations with the $in filter; the dev dry run
is still equivalent to the current v1 slot through the gateway's parse_slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Orchestrator now writes the slot v2 (contract C1) the OPC Gateway reads:
one entry per (server, address) with a names[] list, so an address read by N
pipelines is read once and written to N collections (fixes "one name per
address", where the second model's scouter discarded everything).
- O1: load connectors (opc_ua, Active) and connector_tags (Read, Active) in
parallel with the current loads. Queries are optional in the input with
defaults in code (D9); empty or missing collections load nothing. The loads
sit behind workflow.patched('SIENTIAPDE-1974-tag-database'), so a run in
flight during a deploy replays without them.
- O2: gather_read_tags groups by (server, address). A read tag with tag_id
resolves address/connector from connector_tags and the server from connectors
(RF3); without tag_id it keeps server_id + tag_address + opc-servers (RF4),
so every environment without a tag database keeps collecting (P4).
- O3/SC1: resolve_tag_names is the single source of tag names for both the
scouter's model_tags and the slot's names[].
- O4: build_tag_config emits server_id/name/url/server_uri/use_security/
subscription_period_ms + tags{frequency, active_read, names}; no certificate
paths (the gateway takes the certificate from MinIO). frequency = lowest
among pipelines; subscription period = connector value, else min/2 as before;
active_read off unless a pipeline enables it (lowest cycle_multiple, D3).
- O5: process_slots splits by address (names of an address never split) and
notifies, once per run, unknown tag_id, missing/inactive connector and
unknown opc-servers server_id.
- O6: create_slot_config writes only new or changed slots; the slot is built
in a stable order, so unchanged pipelines do not make the gateway
re-subscribe (and the "Inserted slots" report stops firing every run).
- O7: unit tests for multi-name, repeated address, unknown tag_id, inactive
connector, mixed tags, server-name clash, active_read, slot division,
stability, SC1 and the replay path (184 tests).
- O8: README documents the slot v2 contract and the new optional inputs.
- test_email.test_close: count only the instance under test; an Email left by
another test can be garbage-collected (Email.__del__ -> close) while the
patch is active, which the new tests' allocation pattern exposed.
Dry run with the dev data (scouter-sin-approx): the v2 slot is equivalent to
the current v1 slot as seen by the gateway's parse_slot (same server, addresses,
collections, names, frequency and subscription period).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- NOTIFICATION_QUEUE_BATCH_SIZE default reverted to 0 (unlimited) -- the
existing behavior before this ticket, so deployments must opt in to the
cap rather than having it silently applied at 5000
- requirements-local.txt: bump sientia_do pin to 1.12.3 to match requirements.txt
- Trim the load_latest_data docstring addition so it reads as documentation,
not a changelog entry
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A cold start or notification storm had no backstop: load_latest_data
fetched the entire matching backlog in one query. Confirmed live during
SIENTIAPDE-2012's cluster verification -- an empty Redis cursor plus a
real recurring error let a single run's query exceed Temporal's 4MB
message limit and get stuck retrying forever.
Threads a new notification_queue_batch_size config (env var
NOTIFICATION_QUEUE_BATCH_SIZE, default 5000) through to the repository's
find(), hard cap per run, no drain loop -- the incremental
last_data_timestamp cursor picks up any remainder next run, per the
parent ticket's design decision. Sorts by timestamp ascending so a
capped run always returns the oldest pending documents first; without
that, a capped batch has no guarantee of hitting the oldest documents,
and the cursor would advance past whatever it skipped.
Requires sientia_do>=1.13.0 (adds limit/sort params to
MongoDBRepository.find in both the sync and async client) -- bumped the
pin here per the parent ticket's decision to do it in the same PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
Fix error model retrieval in notification_generator.ipynb and add on_conflict parameter to common_config in orchestrator_functions.py for improved error handling.
Update version in init_orchestration.ipynb to 3.11.14 and enhance README.md with detailed architecture principles, execution flows, and improved filtering mechanisms in workflows. Refactor orchestrator_functions.py for clarity in filter configuration and add support for PI Web API output configuration in predictions_batch. Update tests to reflect new configurations.
Refactor tag processing in pi_web_api_scouter function to improve clarity by renaming variables and enhancing readability of the configuration handling.
Refactor values.yaml to comment out email configuration settings for improved security and clarity. Update orchestrator_functions.py to enhance tag processing by iterating over items in read_tags, improving code readability.
Refactor gather_read_tags function to remove logger parameter and associated logging statements, streamlining the function for improved clarity and focus on processing pipelines.
Update GITHUB_BRANCH to feature/SIENTIAPDE-1445 and enhance formatters with a new registry for workflow types, improving configuration management. Refactor process_schedules to utilize the new registry and add error handling for unsupported workflow types. Update tests to validate new functionality and ensure proper integration.
SIENTIAPDE-1273 Add base_scouter and pi_web_api_scouter functions to orchestrator utilities for enhanced configuration management. Update tests to validate new functionalities and ensure proper integration with existing workflows.
SIENTIAPDE-1273 Add 'save_transform' and 'transform_table_name' parameters to predictions_batch function for enhanced data processing capabilities. Update corresponding test cases to reflect these changes.
Enhance orchestrator utilities and activities with improved documentation and functionality. Update README to reflect new orchestrator functions for drift and simple metrics workflows. Refactor email, formatters, mongo_db, slot_manager, and temporal_manager activities to include detailed docstrings and improve clarity on input parameters and return values. Ensure consistent metadata handling across activities for better logging and tracking.
SIENTIAPDE-1273 Refactor drift function in orchestrator utilities for improved configuration and update formatters to include drift import. Enhance tests to validate new configurations and ensure integration with existing workflows.
Implement simple metrics configuration in formatters and orchestrator utilities. Update tests to validate new simple metrics functionality and ensure proper integration with existing workflows.
Implement drift monitoring configuration and update formatters to support drift workflows. Add drift function to orchestrator utilities and enhance tests for drift functionality.
SIENTIAPDE-1312: Refactor build_tag_config to accept a list of tags and return notifications for missing servers. Update Formatters to handle notifications during slot configuration. Enhance tests to validate new behavior and error reporting.
SIENTIAPDE-1312: Update build_tag_config to ensure minimum frequency defaults to 1000 and convert frequency values to integers for accurate calculations of subscription_period_ms.
SIENTIAPDE-1312: Update build_tag_config to calculate subscription_period_ms based on minimum frequency from tags. Enhance test_build_tag_config to include frequency and validate subscription_period_ms.
Update sientia-dataops-library dependency to version 1.4.7 and enhance TemporalManager to support offset configuration for scheduling. Added offset handling in common configuration and updated related tests to validate new functionality.
Enhance README and code documentation for clarity and structure
- Added a comprehensive Table of Contents to the README for easier navigation.
- Introduced a new section on Code Quality & Validation, detailing the validation workflow and tools used.
- Improved descriptions and consistency in docstrings across various modules, including metrics, formatters, and MongoDB activities.
- Clarified function return types and arguments in the MongoDB and TemporalManager classes for better understanding.
Enhance type hinting and code structure in orchestrator activities
- Added type hints for better code clarity in `formatters.py`, `mongo_db.py`, `temporal_manager.py`, and `email_builder.py`.
- Updated `.gitignore` to include `coverage.xml` for improved coverage reporting.
- Improved readability by restructuring variable declarations and method signatures.
Refactor orchestrator activities and configuration files
- Removed unused Temporal timeout environment variables from values.yaml.
- Reorganized import statements in various activity files for better readability.
- Updated logging messages to use consistent formatting across activities.
- Enhanced test cases to ensure proper initialization and shutdown of orchestrator activities.
- Improved overall code structure and readability by applying consistent formatting and style adjustments.
SIENTIAPDE-1231: Add timeout configuration for Temporal tasks
- Introduced new timeout parameters in values.yaml for task execution.
- Updated TemporalManager to utilize execution and task timeout settings in seconds.
- Enhanced orchestrator_functions.py to include default timeout values for better orchestration control.
Update orchestration configuration and execution parameters
- Adjusted execution counts in Jupyter notebook cells for better tracking.
- Updated schedule name in the orchestration configuration to align with new requirements.
- Modified `values.yaml` to change the GitHub branch for improved versioning.
- Enhanced `Activities` and `TemporalManager` classes to include timeout parameters for task execution.
- Updated connector configuration to include new timeout settings for better orchestration control.
SIENTIAPDE-1222 Refactor server configuration handling in orchestrator_functions.py and update related tests
- Removed the security_spec field from server configuration in build_tag_config to streamline the configuration process.
- Updated test cases in test_formatters.py and test_orchestrator_functions.py to reflect the removal of security_spec, ensuring alignment with the new configuration structure and improving test accuracy.
Enhance OPC server configuration in orchestrator_functions.py
- Added optional fields for certificate paths in the build_tag_config function to improve security configuration flexibility.
Refactor orchestration and email handling
- Updated the orchestration queries to streamline model lookups and added conditions for active models.
- Changed the entry point in the local run script to use the worker module.
- Removed the deprecated samples.json file.
- Enhanced the Email activity to skip sending if the SMTP server is not configured.
- Added timestamp field handling in MongoDB activities for better data management.
- Updated connectors configuration to allow for a None SMTP server.
- Improved the common configuration function to include model configuration details.
Update environment variables in values.yaml and enhance predictions_batch function in orchestrator_functions.py
- Changed GITHUB_BRANCH to reflect the new task for adjusting laborious processes to capture timestamps from MLflow.
- Added a new filter configuration for handling empty data in the predictions_batch function to improve error management.
Remove deprecated files and configurations
- Deleted coverage.sh, Dockerfile, docker-compose.yml, email.html, input_sample.json, and Makefile as part of the cleanup process.
- Updated README.md to reflect the removal of these components and provide a clearer overview of the project structure and functionality.
- Added module-level docstrings to orchestrator, activities, and workflows for improved documentation and clarity on system architecture.
SIENTIAPDE-1205: Enhance predictions_batch function and update test cases
- Added 'predictions_storage_policy' to the predictions_batch function for improved configuration handling.
- Updated test cases to include 'predictions_storage_policy' in expected results, ensuring consistency in testing.
chore: update image tag and enhance datetime column handling in orchestrator functions
- Updated the image tag in values.yaml from 0.4.2 to 0.4.4 for the latest version.
- Added support for datetime_columns in minimal_retrain and predictions_batch functions to improve timestamp handling in orchestrator functions.
refactor: update date handling and remove deprecated constants
- Replaced the usage of datetime.now() with a new now() function from sientia_do.temporal.constants for consistent timestamp handling across multiple files.
- Updated the date format from DEFAULT_DATE_FORMAT to DATETIME_FORMAT_MS_WITH_TZ in various activities to ensure uniformity in timestamp formatting.
- Removed the obsolete patterns.py file that contained the DEFAULT_DATE_FORMAT constant.
chore: update dependencies and refactor logger imports
- Updated the sientia-dataops-library dependency version from 1.3.8 to 1.4.1 in requirements.txt.
- Changed image tag in values.yaml from 0.4.0 to 0.4.2.
- Refactored logger imports in multiple files to use the new observability module instead of the temporal module.
- Updated retry policy imports in various workflow files to reflect the new structure.
refactor: enhance shutdown procedures and documentation across activities
- Added shutdown methods to MongoDB and Email classes to ensure proper resource cleanup.
- Updated docstrings for shutdown methods to clarify their purpose and functionality.
- Enhanced documentation for various methods across multiple classes, improving clarity on parameters and return values.
- Improved the main function and other utility functions with detailed docstrings for better understanding and maintainability.
refactor: streamline email generation and enhance notification handling
- Updated email HTML structure to improve clarity and presentation of notifications.
- Enhanced notification generator to better manage execution counts and output formatting.
- Refactored Email class to improve error handling and logging during email sending.
- Adjusted EmailBuilder to efficiently process error, warning, and info models.
- Improved test coverage for email and notification functionalities to ensure reliability.
