SIENTIAPDE-1974: merge main (0.8.2) into the tag database feature
Brings release/SIENTIAPDE-1974 (slot v2) up to date with main (R1).
Conflicts: both sides kept in orchestrator_functions.py imports, test
imports and README; version follows main (bump is automated on merge).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code review (code-review skill, xhigh) of efa0d94: 8 findings, 6 fixed here,
2 documented (decisions for the team).
Fixed:
- The tag database is read only when a scouter read tag carries a tag_id, and
only the referenced tags ($match tag_id $in, int and text forms), through
aggregations that project only the fields the slot uses. Environments without
a tag database, or whose pipelines do not use tag_id, never query it (P4), and
other attributes (e.g. an ObjectId, which the Temporal JSON converter rejects)
never reach a payload. Fresh aggregation lists are passed, because
aggregate_documents_in_mongodb appends a stage to the list it gets.
- Incomplete connectors / connector_tags documents are left out of the indexes
(their read tags are notified as not found) instead of raising KeyError and
failing the whole run.
- frequency is stored as int, so 15000 vs '15000' across pipelines no longer
makes the slot depend on the pipeline order (O6).
- A missing, null or empty tag_id means "no tag database": an empty string no
longer drops a legacy read tag.
- A read tag without an integer frequency (or, without tag_id, without
server_id / tag_address / tag_name) is skipped and notified
(ORCHESTRATOR_INVALID_READ_TAG_DURING_SLOT_CONFIGURATION) instead of failing
the slot build.
- Server slot keys can no longer collide (name:server_id is extended until free).
Documented in the README:
- The gateway's plan B (READ_ENGINE=legacy) reads v1 only: switching to it after
this deploy also requires rolling the Orchestrator back.
- Renaming a tag in connector_tags does not redeploy the scouter (schedules
update on the pipeline's updated_at).
Validation: 193 unit tests; the real workflow on a local Temporal test server
(sandbox on) with the real Formatters activities and the dev data: slot v2
written without querying the tag database, a second run inserts nothing, and a
tag_id pipeline queries both aggregations with the $in filter; the dev dry run
is still equivalent to the current v1 slot through the gateway's parse_slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Orchestrator now writes the slot v2 (contract C1) the OPC Gateway reads:
one entry per (server, address) with a names[] list, so an address read by N
pipelines is read once and written to N collections (fixes "one name per
address", where the second model's scouter discarded everything).
- O1: load connectors (opc_ua, Active) and connector_tags (Read, Active) in
parallel with the current loads. Queries are optional in the input with
defaults in code (D9); empty or missing collections load nothing. The loads
sit behind workflow.patched('SIENTIAPDE-1974-tag-database'), so a run in
flight during a deploy replays without them.
- O2: gather_read_tags groups by (server, address). A read tag with tag_id
resolves address/connector from connector_tags and the server from connectors
(RF3); without tag_id it keeps server_id + tag_address + opc-servers (RF4),
so every environment without a tag database keeps collecting (P4).
- O3/SC1: resolve_tag_names is the single source of tag names for both the
scouter's model_tags and the slot's names[].
- O4: build_tag_config emits server_id/name/url/server_uri/use_security/
subscription_period_ms + tags{frequency, active_read, names}; no certificate
paths (the gateway takes the certificate from MinIO). frequency = lowest
among pipelines; subscription period = connector value, else min/2 as before;
active_read off unless a pipeline enables it (lowest cycle_multiple, D3).
- O5: process_slots splits by address (names of an address never split) and
notifies, once per run, unknown tag_id, missing/inactive connector and
unknown opc-servers server_id.
- O6: create_slot_config writes only new or changed slots; the slot is built
in a stable order, so unchanged pipelines do not make the gateway
re-subscribe (and the "Inserted slots" report stops firing every run).
- O7: unit tests for multi-name, repeated address, unknown tag_id, inactive
connector, mixed tags, server-name clash, active_read, slot division,
stability, SC1 and the replay path (184 tests).
- O8: README documents the slot v2 contract and the new optional inputs.
- test_email.test_close: count only the instance under test; an Email left by
another test can be garbage-collected (Email.__del__ -> close) while the
patch is active, which the new tests' allocation pattern exposed.
Dry run with the dev data (scouter-sin-approx): the v2 slot is equivalent to
the current v1 slot as seen by the gateway's parse_slot (same server, addresses,
collections, names, frequency and subscription period).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enhance orchestration configuration and documentation. Added `RUNTIME` variable to `.env.example`, updated `.gitignore` to exclude `openspec/` and `.cursor/`, and modified `README.md` to clarify queue naming conventions and runtime handling. Refactored activities to use synchronous database and email handling, improving performance and consistency. Updated test cases to reflect these changes and ensure compatibility with new activity definitions.
Update version in init_orchestration.ipynb to 3.11.14 and enhance README.md with detailed architecture principles, execution flows, and improved filtering mechanisms in workflows. Refactor orchestrator_functions.py for clarity in filter configuration and add support for PI Web API output configuration in predictions_batch. Update tests to reflect new configurations.
Enhance orchestrator utilities and activities with improved documentation and functionality. Update README to reflect new orchestrator functions for drift and simple metrics workflows. Refactor email, formatters, mongo_db, slot_manager, and temporal_manager activities to include detailed docstrings and improve clarity on input parameters and return values. Ensure consistent metadata handling across activities for better logging and tracking.
Remove unused Couchbase configurations and related code. Refactor Activities class to eliminate Couchbase dependency, updating initialization and shutdown methods. Update README and tests to reflect these changes. Upgrade sientia-dataops-library dependency version in requirements.txt.
Enhance README and code documentation for clarity and structure
- Added a comprehensive Table of Contents to the README for easier navigation.
- Introduced a new section on Code Quality & Validation, detailing the validation workflow and tools used.
- Improved descriptions and consistency in docstrings across various modules, including metrics, formatters, and MongoDB activities.
- Clarified function return types and arguments in the MongoDB and TemporalManager classes for better understanding.
Enhance README and activities with advanced notification filtering features
- Updated README to include new intelligent notification filtering capabilities, including TTL-based duplicate prevention and ignore lists.
- Enhanced `Formatters` activity to support notification filtering for scheduled reports and added detailed filtering logic.
- Improved `SlotManager` to implement advanced notification filtering with persistent alert detection and group-based filtering.
- Updated `Alerts` workflow to incorporate intelligent filtering for real-time error notifications, ensuring efficient alert delivery.
Update README.md to enhance system architecture documentation
- Removed outdated system overview diagram and replaced it with updated architecture diagrams for workflows, including Data Loading, Processing, and Notifications.
- Added detailed architecture sections for the Alerts and Reports workflows, including input parameters and flowcharts to illustrate processes.
- Improved clarity and organization of the README to better reflect the current system design and functionality.
Remove deprecated files and configurations
- Deleted coverage.sh, Dockerfile, docker-compose.yml, email.html, input_sample.json, and Makefile as part of the cleanup process.
- Updated README.md to reflect the removal of these components and provide a clearer overview of the project structure and functionality.
- Added module-level docstrings to orchestrator, activities, and workflows for improved documentation and clarity on system architecture.
docs: update README.md to provide comprehensive overview of SIENTIA DataOps Orchestrator Temporal
- Added detailed project overview, goals, and architecture sections.
- Documented main workflows including orchestrator, alerts, and reports workflows with input parameters and functionalities.
- Included environment variable configurations for Redis, MongoDB, PostgreSQL, Couchbase, email, Temporal, application, and Kafka.
- Described deployment strategies and monitoring capabilities, including Prometheus metrics.
- Enhanced usage instructions for triggering workflows and logging operations.
Add unit tests for connectors configuration, logger, workflows, and predictions batch
- Implement tests for MLflow, OPC, and Postgres configuration builders to validate environment variable handling and default values.
- Create tests for the logger to ensure default settings and handler configurations are correct.
- Add comprehensive tests for the FormatAndExportPrediction and PredictionProcess workflows, covering various scenarios including path flags and activity execution.
- Introduce tests for the PredictionsBatch workflow to verify the execution of local activities and child workflows.
- Include a values.yaml file for Kubernetes deployment configuration, specifying image details, service account settings, environment variables, and resource limits.
