gitea_admin/sientia-dataops-scouter_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1478

Browse Source 
Enhance PI Web API Scouter Workflow and Documentation

- Added support for PI Web API data ingestion, including real-time and historical data retrieval.
- Implemented timestamp normalization to ensure consistency across records.
- Updated README.md to reflect new features and detailed workflow execution flow.
- Enhanced API class with comprehensive error handling and cleanup operations.
- Improved CoreScouter workflow with early exit conditions and detailed processing steps.
- Updated configuration parameters for better clarity and usability.
This commit is contained in:
vitor-aignosi
2026-01-09 09:49:55 -03:00
parent b75f499436
commit 4352c67f3a
5 changed files with 150 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
scouter/workflow/pi_web_api_scouter.py
View File

@@ -33,26 +33,31 @@ class PIWebAPIScouter:
This method orchestrates the complete data ingestion process from PI Web API:
1. Retrieves tag values from PI Web API using configured WebIds
2. Validates and normalizes the retrieved data
2. Validates and normalizes the retrieved data (timestamps are normalized)
3. Delegates data processing to the CoreScouter workflow
If no data is retrieved from the PI Web API, the workflow exits early without
invoking the CoreScouter workflow.
Args:
input_data (dict[str, Any]): Configuration and parameters for the workflow execution.
Required fields:
- schedule_name (str): Unique identifier for the data collection schedule
- model_name (str): Name of the data model being processed
- model_id (str): Unique identifier for the data model
- schedule_name (str): Unique identifier for the data collection schedule
- endpoint (str): PI Web API endpoint path (e.g., '/streamsets/recorded')
- web_ids (dict[str, str | None]): Mapping of tag names to WebIds
- period (dict[str, str]): Time period configuration with 'start_time'
- api_timeout (int): Request timeout in seconds for PI Web API calls
- max_count (int, optional): Maximum data points per tag. Defaults to 1
- pi_web_api_query (dict[str, Any]): PI Web API query configuration containing:
- endpoint (str): PI Web API endpoint path (e.g., '/streamsets/recorded')
- period (str): Time period configuration (e.g., '*-1d', '*-1h')
- api_timeout (int): Request timeout in seconds for PI Web API calls
- max_count (int, optional): Maximum data points per tag. Defaults to 1
- trigger_laborious (bool): Flag to enable intensive data processing
- filters (dict[str, str]): Data quality filters configuration
- schema (str): Target database schema for data export
- table_name (str): Target table name for data export
- retention_time (int): Data retention period in Redis (seconds)
- model_tags (dict[str, Any]): Tag-specific configuration including:
- model_tags (dict[str, Any]): Tag-specific configuration mapping tag names
to WebIds and processing rules, including:
- webid (str): PI Web API WebId for the tag
- data_range: [min, max] values for data validation
- aggr_function: Aggregation method (avg, mdn, max, min, lts)
- frequency: Data collection frequency in milliseconds

8
scouter/workflow/scouter.py
View File

@@ -32,10 +32,14 @@ class Scouter:
This method orchestrates the complete data ingestion process:
1. Retrieves the last processed timestamp from Redis
2. Loads new data from MongoDB since the last timestamp
3. Updates the last processed timestamp
2. Loads new data from MongoDB since the last timestamp using collection name
format: `raw_{schedule_name}`
3. Updates the last processed timestamp with the most recent data point
4. Delegates data processing to the CoreScouter workflow
If no new data is found in MongoDB, the workflow exits early without updating
the timestamp or invoking the CoreScouter workflow.
Args:
input_data (dict[str, Any]): Configuration and parameters for the workflow execution.
Required fields:

13
scouter/workflow/sub_workflows/core_scouter.py
View File

@@ -23,7 +23,8 @@ class CoreScouter:
- Metrics collection and monitoring
The workflow is designed for high-throughput data processing with configurable
quality gates and aggregation strategies.
quality gates and aggregation strategies. It is typically invoked as a child
workflow by parent workflows such as Scouter or PIWebAPIScouter.
"""
@workflow.run
@@ -35,9 +36,14 @@ class CoreScouter:
1. Data Quality Gate: Applies configurable filters for data validation
2. Data Aggregation: Groups and aggregates data using specified functions
3. Data Grouping: Organizes data by tags and applies retention policies
4. Data Export: Persists processed data to PostgreSQL
4. Data Export: Persists processed data to PostgreSQL with timestamp conversion
5. Metrics Collection: Records processing metrics for monitoring
The workflow implements early exit conditions:
- If held_data is empty after grouping, the workflow exits without exporting
- If data export results in zero or negative affected_rows, the workflow exits
without writing metrics or storing debug packages
Args:
input_data (dict[str, Any]): Complete workflow configuration and data.
Required fields:
@@ -53,6 +59,9 @@ class CoreScouter:
- table_name (str): Target database table
- retention_time (int): Redis data retention period (seconds)
- model_tags (dict[str, Any]): Tag-specific processing rules
- fill_missing_tags (bool): Enable filling of missing tag values
- debug_data_package (bool, optional): Store data packages for debugging.
When True, stores both raw and processed data in MongoDB for debugging
Returns:
None: This workflow processes data but doesn't return results