Sientia DataOps Laborious

The Sientia DataOps Laborious is a Temporal-based workflow application that handles batch predictions and data processing for industrial data. It integrates with MLFlow for model management, PostgreSQL for data storage, and OPC for real-time data output. The module is designed to process data in a reliable and scalable manner using Temporal.io's workflow orchestration capabilities. It's get data from Scouter sinks, process it, make predictions using MLFlow models and generates metrics for the predictions.

Key Features

• Batch predictions using MLFlow models
• Data transformation and preprocessing
• Workflow orchestration using Temporal.io
• Integration with PostgreSQL for data storage
• OPC integration for real-time data output
• Comprehensive error handling and notifications
• Configurable data filters and quality gates
• Scalable deployment architecture

Workflows

Predictions Batch

The main workflow that orchestrates batch predictions. Steps:

• prepare_activity: Prepares the activity with schedule and model information
• load_custom_query: Loads data using a custom query
• prediction_process: Executes the prediction process using the Prediction Process sub-workflow

Workflow inputs:

• schedule_name: The schedule name of the activity
• model_name: The model name of the activity
• model_id: The model id of the activity
• query: The custom query to load data
• schema: The schema of the data
• table_name: The name of the table to process
• input_filters: The filters to be applied during prediction
• mlflow_transform_filters: The filters to be applied during prediction
• mlflow_predict_filters: The filters to be applied during prediction
• model_retention: The model retention period in minutes
• path_priority: The path priority

Prediction Process

Sub-workflow that handles individual prediction processing:

• get_last_timestamp: Gets the last timestamp of the data
• input_gate: Filters input data based on configured rules
• repeat_last_prediction: Repeats the last prediction if the data is empty
• request_transform: Makes predictions using MLFlow models
• mlflow_response_gate: Handles prediction or transform responses and filters
• mlflow_content_gate: Filters transform responses based on configured rules
• request_predict: Makes predictions using MLFlow models
• format_and_export_prediction: Formats and exports predictions using the Format and Export Prediction sub-workflow

Format and Export Prediction

Sub-workflow that handles prediction formatting and export:

• format_prediction: Formats prediction data if path flag is None
• format_default_prediction: Formats default prediction data if path flag is not None
• export_to_postgres: Exports formatted predictions to PostgreSQL
• write_to_opc: Writes predictions to OPC server

Environment variables

• POSTGRES_HOST

• POSTGRES_PORT

• POSTGRES_USER

• POSTGRES_PASSWORD

• POSTGRES_DBNAME

• POSTGRES_MIN_CONNECTIONS

• POSTGRES_MAX_CONNECTIONS

• MLFLOW_HOST

• MLFLOW_PORT

• MLFLOW_USERNAME

• MLFLOW_PASSWORD

• OPC_CONFIG - json string containing the opc configuration for multiple opc servers For single opc server use:

• OPC_URL

• OPC_NAME

• OPC_SERVER_URI

• OPC_CERT_PATH

• OPC_PRIVATE_KEY_PATH

• OPC_SERVER_CERT_PATH

• OPC_RECONNECTION_INTERVAL

• TEMPORAL_HOST

• TEMPORAL_NAMESPACE

Application deployment

The application can be deployed using the following command:

helm upgrade --install sientia-dataops-laborious sientia/sientia-module -n sientia --create-namespace -f ./values.yaml