gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1478

Browse Source 
Enhance README and Codebase with PI Web API Integration

- Updated README.md to include details about PI Web API integration, including configuration and export capabilities.
- Modified Activities class to incorporate PI Web API export operations and error handling.
- Added new API class for handling PI Web API interactions, including writing prediction and confidence data.
- Updated prediction workflows to support PI Web API output configuration.
- Enhanced worker and sub-workflows to include PI Web API in task queues and export processes.
- Improved documentation and error handling for PI Web API connections and configurations.
This commit is contained in:
vitor-aignosi
2026-01-09 09:30:26 -03:00
parent e8b7105e9b
commit 7cfa34a963
7 changed files with 132 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
laborious/activities/activities.py
View File

@@ -24,15 +24,18 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
MLFlow model interactions, data quality validation, and OPC server communications.
The class implements multiple inheritance to combine specialized functionality:
- Postgres: Database operations and data persistence
- Storage: Database operations and data persistence
- MLFlow: Model inference and transformation operations
- Gates: Data quality validation and filtering mechanisms
- OPC: Real-time data export to OPC servers
- ModelMetrics: Model performance metrics and drift detection
- API: PI Web API export operations for industrial systems
Attributes:
postgres_config (dict): PostgreSQL connection configuration
mlflow_config (dict): MLFlow server configuration
opc_config (dict): OPC server configuration
pi_web_api_config (dict): PI Web API server configuration
logger (Logger): Logging and observability instance
notification_handler (NotificationHandler): Notification management instance
"""
@@ -137,6 +140,8 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
This method ensures proper cleanup of all resources including:
- PostgreSQL connection pools
- OPC server connections
- PI Web API client connections
- MLFlow model repositories
- Any other resources that need explicit cleanup
The method should be called before the application terminates to ensure

31
laborious/activities/api.py
View File

@@ -18,10 +18,16 @@ PI_WEB_API_PREDICTION_ERROR_CONFIDENCE = 13
class API(SientiaMonitoring):
"""
PI Web API operations for writing data to PI Web API.
PI Web API operations for writing prediction data to PI Web API.
This class provides Temporal activities for interacting with the PI Web API
to write data to PI Web API.
to write prediction and confidence values to industrial systems. It handles
error scenarios gracefully by setting error confidence values and sending
notifications when write operations fail.
The class implements comprehensive error handling for both prediction and
confidence value writes, ensuring that partial failures are properly
reported and handled.
"""
def __init__(
@@ -66,13 +72,24 @@ class API(SientiaMonitoring):
@activity.defn(name='write_pi_web_api_data')
async def write_pi_web_api_data(self, input_data: dict[str, Any]) -> dict[Any, Any]:
"""
Write data to PI Web API.
Write prediction and confidence data to PI Web API.
This method writes prediction values and confidence scores to PI Web API
using configured web IDs. It handles errors gracefully by setting error
confidence values when prediction writes fail and sending notifications
for both prediction and confidence write errors.
Args:
input_data (dict[str, Any]): The input data. Containing:
- metadata (dict[str, Any]): The metadata.
- pi_web_api_output_config (dict[str, Any]): The PI Web API output configuration.
- data (dict[str, Any]): The data to write.
input_data (dict[str, Any]): The input data containing:
- metadata (dict[str, Any]): Workflow execution metadata
- pi_web_api_output_config (dict[str, Any]): PI Web API configuration with:
- endpoint (str): PI Web API endpoint URL
- prediction_tags (dict[str, str]): Mapping of tag names to web IDs for predictions
- confidence_tags (dict[str, str]): Mapping of tag names to web IDs for confidence
- data (dict[str, Any]): Prediction data, its a dataframe converted to dict.
Returns:
dict[Any, Any]: Data dictionary with potentially modified confidence values
If prediction write fails, prediction_confidence is set to error value (13)
"""
metadata = input_data['metadata']
data = DataFrame(input_data['data'])

1
laborious/worker/worker.py
View File

@@ -7,6 +7,7 @@ prediction and retraining workflows.
The worker supports multiple task queues:
- predictions_batch-queue: Handles batch prediction workflows (heavy workload)
Includes activities for MLFlow, data quality gates, OPC export, PI Web API export, and PostgreSQL
- minimal_retrain-queue: Handles model retraining workflows
- drift-queue: Handles drift detection workflows
- simple_metrics-queue: Handles simple metrics calculation workflows

3
laborious/workflows/predictions_batch.py
View File

@@ -61,7 +61,10 @@ class PredictionsBatch:
- model_retention (int, optional): Model retention period in minutes
- path_priority (list[str]): Decision path priority configuration
- opc_output_config (dict, optional): OPC server export configuration
- pi_web_api_output_config (dict, optional): PI Web API export configuration
- datetime_columns (list[str], optional): Columns to treat as datetime
- save_transform (bool, optional): Whether to save transformed data (default: True)
- prediction_store_policy (str, optional): Data retention policy (default: 'lts:1')
Returns:
None: The workflow completes successfully when the child workflow finishes

12
laborious/workflows/sub_workflows/format_and_export_prediction.py
View File

@@ -26,6 +26,7 @@ class FormatAndExportPrediction:
Export Destinations:
- PostgreSQL Database: Persistent storage with timestamp conversion
- PI Web API: Real-time industrial system integration for prediction and confidence values
- OPC Servers: Real-time industrial system integration
- Prometheus Metrics: Performance monitoring and operational visibility
"""
@@ -38,9 +39,10 @@ class FormatAndExportPrediction:
This method orchestrates the complete data export process by:
1. Determining the appropriate formatting strategy based on path_flag
2. Formatting prediction data according to quality and requirements
3. Exporting data to OPC servers for real-time industrial access
4. Persisting data to PostgreSQL database with comprehensive metadata
5. Recording performance metrics for operational monitoring
3. Exporting data to PI Web API for real-time industrial access (if configured)
4. Exporting data to OPC servers for real-time industrial access (if configured)
5. Persisting data to PostgreSQL database with comprehensive metadata
6. Recording performance metrics for operational monitoring
The method implements flexible formatting strategies:
- Normal predictions: Full data formatting with confidence scores
@@ -61,8 +63,10 @@ class FormatAndExportPrediction:
- model_name (str): Name of the ML model
- schema (str): Database schema for data storage
- table_name (str): Target table for data persistence
- opc_output_config (dict[str, Any]): OPC server export configuration
Optional keys:
- opc_output_config (dict[str, Any]): OPC server export configuration
- pi_web_api_output_config (dict[str, Any]): PI Web API export configuration
Contains endpoint, prediction_tags, and confidence_tags mappings
- transformed_data (dict[str, Any]): Transformed data to export separately
Only processed when path_flag is None
- transform_table_name (str): Target table for transformed data export

19
laborious/workflows/sub_workflows/prediction_process.py
View File

@@ -66,7 +66,10 @@ class PredictionProcess:
- mlflow_predict_filters (dict): MLFlow prediction filters
- model_retention (int): Model retention period in minutes
- path_priority (list[str]): Decision path priority configuration
- opc_output_config (dict): OPC server export configuration
- opc_output_config (dict, optional): OPC server export configuration
- pi_web_api_output_config (dict, optional): PI Web API export configuration
- save_transform (bool, optional): Whether to save transformed data (default: True)
- prediction_store_policy (str, optional): Data retention policy (default: 'lts:1')
Returns:
None: The workflow completes successfully when export workflow finishes
@@ -235,7 +238,17 @@ class PredictionProcess:
Args:
data: Input data for processing
path_flag: Path decision from filter (STOP, CONTINUE, REPEAT)
input_data: Complete workflow input configuration
input_data: Complete workflow input configuration including:
- metadata (dict): Workflow execution metadata
- schema (str): Database schema
- table_name (str): Target table for predictions
- transform_table_name (str): Target table for transformed data
- model_id (str): ML model identifier
- model_name (str): ML model name
- model_config (dict, optional): Model configuration
- opc_output_config (dict, optional): OPC server export configuration
- pi_web_api_output_config (dict, optional): PI Web API export configuration
- prediction_store_policy (str, optional): Data retention policy
confidence: Confidence level from filter validation
last_timestamp: Last processed timestamp
comment: Additional information about the filter result
@@ -245,7 +258,7 @@ class PredictionProcess:
Path Handling:
- STOP: Terminates workflow execution
- CONTINUE: Proceeds with normal processing
- CONTINUE: Delegates to FormatAndExportPrediction workflow with current data
- REPEAT: Repeats last prediction if available
"""
metadata = input_data['metadata']