Merge pull request #35 from Aignosi/feature/SIENTIAPDE-1478

Feature: PI Web API Integration, Enhanced PredictionsBatch E2E Tests, and System Refinements

.github/workflows/release.yml

@@ -21,5 +21,5 @@ jobs:
     permissions: write-all
     with:
       project_name: 'laborious'
-      version: ${{ github.event.inputs.version || '' }}
+      release_version: ${{ github.event.inputs.version || '' }}
     secrets: inherit

README.md

@@ -1,6 +1,6 @@
 # Sientia DataOps Laborious
 
-A comprehensive, Temporal-based ML orchestration system for industrial data processing and model inference. Laborious delivers enterprise-grade batch prediction, model management, optional real-time export (OPC), and automated retraining with strong data quality validation and observability.
+A comprehensive, Temporal-based ML orchestration system for industrial data processing and model inference. Laborious delivers enterprise-grade batch prediction, model management, optional real-time export (OPC and PI Web API), and automated retraining with strong data quality validation and observability.
 
 
 ## 📑 Table of Contents
@@ -71,7 +71,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
 - **Temporal Workflow Orchestration**: Robust workflow management with retries and fault tolerance
 - **Data Quality Gates**: Configurable filtering for input data and MLFlow API responses
 - **Multi-Model Support**: Flexible model management with retention and versioning
-- **Optional Real-time Export**: PostgreSQL persistence and OPC server integration for industrial systems
+- **Optional Real-time Export**: PostgreSQL persistence, OPC server integration, and PI Web API integration for industrial systems
 - **Comprehensive Monitoring**: Prometheus metrics and structured logging for observability
 
 ### Advanced Capabilities
@@ -140,6 +140,9 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
   - Model retraining and production updates
   - Reference data retrieval from MLflow Model Registry
 - `opc.py`: OPC UA export to industrial systems (optional)
+- `api.py`: PI Web API export operations (optional)
+  - Prediction and confidence data writing to PI Web API
+  - Error handling and notification integration
 - `activities.py`: Aggregates activity interfaces
 
 #### **Data Services (`laborious/utils/`)**
@@ -154,7 +157,7 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
 ```
 Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform →
 MLFlow Prediction → Response Validation → Format & Export
-  ├─→ Predictions → PostgreSQL [+ OPC]
+  ├─→ Predictions → PostgreSQL [+ OPC] [+ PI Web API]
   └─→ Transformed Data → PostgreSQL (optional)
 ```
 
@@ -170,6 +173,7 @@ Production Update → Notification & Monitoring
 - **MLFlow API Authentication**: Username/password
 - **Database Security**: Encrypted connections and credential management
 - **OPC Certificates** (if enabled): Client/server certs
+- **PI Web API Authentication**: Bearer token or basic authentication
 - **Kubernetes Secrets**: Secure secret storage
 
 #### **Network Security**
@@ -230,6 +234,11 @@ The **PredictionsBatch** workflow is the main entry point for batch prediction p
   "opc_output_config": {
     "server_id": "opc_server_1",
     "tags": ["prediction_output"]
+  },
+  "pi_web_api_output_config": {
+    "endpoint": "https://pi-server.com/piwebapi",
+    "prediction_tags": {"tag1": "web_id_1"},
+    "confidence_tags": {"tag2": "web_id_2"}
   }
 }
 ```
@@ -299,7 +308,12 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M
   },
   "model_retention": 60,
   "path_priority": ["STOP", "CONTINUE", "REPEAT"],
-  "opc_output_config": {...}
+  "opc_output_config": {...},
+  "pi_web_api_output_config": {
+    "endpoint": "https://pi-server.com/piwebapi",
+    "prediction_tags": {"tag1": "web_id_1"},
+    "confidence_tags": {"tag2": "web_id_2"}
+  }
 }
 ```
 
@@ -322,19 +336,21 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an
 - **Data Formatting**: Formats prediction data for different output destinations
 - **PostgreSQL Export**: Persists predictions to database with metrics
 - **OPC Integration**: Writes predictions to OPC servers for real-time access
+- **PI Web API Integration**: Writes predictions and confidence to PI Web API for industrial systems
 - **Metrics Recording**: Tracks export operations and performance metrics
 
 #### Execution Flow
 1. **Path Decision**: Determines formatting path based on configuration
 2. **Data Formatting**: Formats prediction data for specific output requirements
 3. **Transformed Data Processing**: Optionally formats and exports transformed data separately
-4. **PostgreSQL Export**: Writes formatted predictions to database
+4. **PI Web API Export**: Writes predictions and confidence to PI Web API (if configured)
-5. **OPC Export**: Writes predictions to OPC servers
+5. **OPC Export**: Writes predictions to OPC servers (if configured)
-6. **Metrics Recording**: Records export performance and success metrics
+6. **PostgreSQL Export**: Writes formatted predictions to database
+7. **Metrics Recording**: Records export performance and success metrics
 
 #### Key Features
 - **Flexible Formatting**: Configurable output formats for different destinations
-- **Multi-Destination Export**: PostgreSQL and OPC server integration
+- **Multi-Destination Export**: PostgreSQL, OPC server, and PI Web API integration
 - **Transformed Data Export**: Optional separate export of MLFlow transformed data
 - **Performance Monitoring**: Comprehensive metrics for export operations
 - **Error Handling**: Robust error handling with notification integration
@@ -342,13 +358,14 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an
 #### Architecture Diagram
 ```mermaid
 flowchart LR
-    A[1. format_prediction/format_default_prediction] --> B[2. format_transformed_data] --> C[3. write_opc_data] --> D[4. export_data_to_postgres] --> E[5. write_metrics]
+    A[1. format_prediction/format_default_prediction] --> B[2. format_transformed_data] --> C[3. write_pi_web_api_data] --> D[4. write_opc_data] --> E[5. export_data_to_postgres] --> F[6. write_metrics]
     
     A -.-> Format[Data Formatting]
     B -.-> Transform[Transformed Data]
-    C -.-> OPC[OPC Servers]
+    C -.-> PIWebAPI[PI Web API]
-    D -.-> PostgreSQL[(PostgreSQL)]
+    D -.-> OPC[OPC Servers]
-    E -.-> Prometheus[Prometheus]
+    E -.-> PostgreSQL[(PostgreSQL)]
+    F -.-> Prometheus[Prometheus]
 ```
 
 #### Transformed Data Export
@@ -395,6 +412,7 @@ flowchart LR
 - MinIO object storage (for MLFlow artifacts)
 - MongoDB server (for notifications)
 - OPC server(s) if using OPC export
+- PI Web API server if using PI Web API export
 
 **Note**: External dependencies must be available either through:
 - Kubernetes cluster deployment
@@ -685,6 +703,9 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
 | `OPC_PRIVATE_KEY_PATH` | OPC private key path | `None` | No |
 | `OPC_SERVER_CERT_PATH` | OPC server certificate path | `None` | No |
 | `OPC_RECONNECTION_INTERVAL` | OPC reconnection interval (ms) | `120` | No |
+| `PI_WEB_API_BASE_URL` | PI Web API server base URL | `None` | No |
+| `PI_WEB_API_AUTH_TYPE` | PI Web API authentication type (basic/bearer) | `None` | No |
+| `PI_WEB_API_AUTH_TOKEN` | PI Web API authentication token | `None` | No |
 | `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
 | `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
 | `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
@@ -735,6 +756,37 @@ For single OPC server, use individual environment variables:
 - `OPC_SERVER_CERT_PATH`
 - `OPC_RECONNECTION_INTERVAL`
 
+### PI Web API Configuration
+
+PI Web API configuration is built from environment variables using the `build_api_config` function from `sientia_do.connectors_config`. The configuration includes:
+
+- `PI_WEB_API_BASE_URL`: Base URL of the PI Web API server
+- `PI_WEB_API_AUTH_TYPE`: Authentication type ('basic' or 'bearer')
+- `PI_WEB_API_AUTH_TOKEN`: Authentication token for API access
+
+The PI Web API export is optional and can be configured per workflow through the `pi_web_api_output_config` parameter:
+
+```json
+{
+  "pi_web_api_output_config": {
+    "endpoint": "https://pi-server.com/piwebapi",
+    "prediction_tags": {
+      "tag1": "web_id_1",
+      "tag2": "web_id_2"
+    },
+    "confidence_tags": {
+      "tag3": "web_id_3",
+      "tag4": "web_id_4"
+    }
+  }
+}
+```
+
+Where:
+- `endpoint`: PI Web API endpoint URL
+- `prediction_tags`: Dictionary mapping tag names to web IDs for prediction values
+- `confidence_tags`: Dictionary mapping tag names to web IDs for confidence values
+
 ### Workflow Configuration
 
 MongoDB pipeline configuration:
@@ -833,7 +885,8 @@ laborious/
 │   ├── activities.py       # Main activities orchestrator
 │   ├── gates.py            # Data quality gates and filtering
 │   ├── mlflow.py           # MLFlow model operations
-│   └── opc.py              # OPC server operations
+│   ├── opc.py              # OPC server operations
+│   └── api.py              # PI Web API operations
 ├── workflows/               # Temporal workflow definitions
 │   ├── predictions_batch.py # Main batch prediction workflow
 │   ├── minimal_retrain.py  # Model retraining workflow
@@ -886,7 +939,14 @@ laborious/
    - Check certificate and key file paths
    - Review OPC server logs for connection issues
 
-5. **Workflow Execution Failures**
+5. **PI Web API Connection Failures**
+   - Verify PI Web API server is accessible
+   - Check authentication credentials and token validity
+   - Verify web IDs exist and have write permissions
+   - Review PI Web API server logs for connection issues
+   - Check notification system for error details
+
+6. **Workflow Execution Failures**
    - Review activity error logs and notifications
    - Check data quality filter configurations
    - Verify input data format and required fields

e2e/__init__.py

@@ -0,0 +1,3 @@
+"""
+End-to-end tests for laborious temporal workflows.
+"""

e2e/conftest.py

@@ -0,0 +1,452 @@
+"""
+Pytest configuration and fixtures for E2E tests.
+"""
+
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pandas as pd
+import pytest
+import pytest_asyncio
+from sqlalchemy import create_engine, text
+from testcontainers.postgres import PostgresContainer
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from laborious.activities.activities import Activities
+from laborious.workflows.predictions_batch import PredictionsBatch
+from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
+from laborious.workflows.sub_workflows.format_and_export_prediction import FormatAndExportPrediction
+from sientia_do.notifications.handlers import CoreNotificationHandler
+from sientia_do.observability.logger import Logger
+from sientia_do.observability.metrics_controller import MetricsController
+
+# Test constants
+TEST_MONGODB_CONNECTION_STRING = 'mongodb://localhost:27017'
+TEST_DATABASE_NAME = 'test_db'
+
+
+@pytest_asyncio.fixture(scope='session')
+def postgres_container():
+    """
+    Create a PostgreSQL container using testcontainers.
+    
+    This fixture creates a real PostgreSQL database in a Docker container
+    that will be used for all tests in the session.
+    """
+    postgres = PostgresContainer('postgres:15')
+    postgres.start()
+    yield postgres
+    postgres.stop()
+
+
+@pytest_asyncio.fixture
+def postgres_engine(postgres_container):
+    """
+    Create SQLAlchemy engine for PostgreSQL test database.
+    
+    This fixture creates a connection to the PostgreSQL container
+    created by the postgres_container fixture.
+    """
+    engine = create_engine(postgres_container.get_connection_url())
+    
+    yield engine
+    
+    engine.dispose()
+
+
+def _create_schema_and_tables(engine):
+    """
+    Helper function to create schema and tables in the given engine.
+    
+    Creates predictions_schema with:
+    - laborious_data: Input data table for queries
+    - predictions: Output predictions table
+    - transformed_data: Output transformed data table
+    """
+    # Use begin() to ensure transaction is properly committed
+    with engine.begin() as conn:
+        # Create predictions_schema
+        conn.execute(text("CREATE SCHEMA IF NOT EXISTS predictions_schema"))
+        
+        # Create laborious_data table (input data from sensors)
+        create_laborious_data_sql = """
+            CREATE TABLE IF NOT EXISTS predictions_schema.laborious_data (
+                id SERIAL NOT NULL,
+                model_id int4 NOT NULL,
+                variable text NOT NULL,
+                value numeric NULL,
+                "timestamp" timestamptz NOT NULL,
+                created_at timestamptz NOT NULL,
+                PRIMARY KEY (id)
+            );
+        """
+        conn.execute(text(create_laborious_data_sql))
+        
+        # Create predictions table
+        create_predictions_sql = """
+            CREATE TABLE if not exists predictions_schema.predictions (
+                id SERIAL NOT NULL ,
+                model_id int4 NOT NULL,
+                prediction numeric NULL,
+                prediction_confidence numeric NOT NULL,
+                response_time numeric NOT NULL,
+                prediction_status text NOT NULL,
+                "timestamp" timestamptz NOT NULL,
+                created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+                "comments" text NULL,
+                PRIMARY KEY (id, created_at)
+            );
+        """
+        conn.execute(text(create_predictions_sql))
+        
+        # Create transformed_data table
+        create_transformed_sql = """
+            CREATE TABLE IF NOT EXISTS predictions_schema.transformed_data (
+                id SERIAL NOT NULL,
+                model_id int4 NOT NULL,
+                variable text NOT NULL,
+                value numeric NULL,
+                "timestamp" timestamptz NOT NULL,
+                created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
+                PRIMARY KEY (id)
+            );
+        """
+        conn.execute(text(create_transformed_sql))
+
+
+@pytest_asyncio.fixture(autouse=True)
+def setup_postgres_schema_and_tables(postgres_engine):
+    """
+    Automatically create necessary schema and tables before each test.
+    
+    This fixture runs automatically (autouse=True) and ensures
+    that the predictions_schema and tables exist with the correct structure.
+    """
+    _create_schema_and_tables(postgres_engine)
+    yield
+
+
+@pytest_asyncio.fixture
+def mock_logger():
+    """Mock logger for testing."""
+    def message(message):
+        print(f"[LOG] {message}")
+    def custom_message(message, _metadata={}):
+        print(f"[LOG] {message}")
+    logger = MagicMock()
+    logger.info = MagicMock(
+        side_effect=message
+    )
+    logger.debug = MagicMock(
+        side_effect=message
+    )
+    logger.error = MagicMock(
+        side_effect=message
+    )
+    logger.warning = MagicMock(
+        side_effect=message
+    )
+    logger.custom_info = MagicMock(
+        side_effect=custom_message
+    )
+    logger.custom_debug = MagicMock(
+        side_effect=custom_message
+    )
+    logger.custom_error = MagicMock(
+        side_effect=custom_message
+    )
+    logger.custom_warning = MagicMock(
+        side_effect=custom_message
+    )
+    return logger
+
+
+@pytest_asyncio.fixture
+def mock_mongo_client():
+    """
+    Mock MongoDB client to avoid real connections.
+    
+    This fixture mocks the pymongo.MongoClient used by CoreNotificationHandler,
+    allowing us to use a real NotificationHandler instance without connecting to MongoDB.
+    """
+    mock_client = MagicMock()
+    mock_db = MagicMock()
+    mock_collection = MagicMock()
+    
+    # Configure the mock chain: client[database] -> db[collection] -> collection
+    mock_client.__getitem__.return_value = mock_db
+    mock_db.__getitem__.return_value = mock_collection
+    
+    # Mock server_info() to avoid connection attempts
+    mock_client.server_info = MagicMock()
+    
+    # Mock insert_one for notifications
+    mock_collection.insert_one = MagicMock()
+    
+    return mock_client
+
+
+@pytest_asyncio.fixture
+def notification_handler(mock_logger, mock_mongo_client):
+    """
+    Create a real NotificationHandler instance with mocked MongoDB client.
+    
+    This fixture creates a real CoreNotificationHandler instance but mocks
+    the underlying MongoDB connection to avoid real database connections.
+    """
+    # Patch MongoClient where it's imported in the handlers module
+    with patch('sientia_do.notifications.handlers.MongoClient', return_value=mock_mongo_client):
+        handler = CoreNotificationHandler(
+            connection_string=TEST_MONGODB_CONNECTION_STRING,
+            database=TEST_DATABASE_NAME,
+            logger=mock_logger,
+            project_name='laborious',
+        )
+        yield handler
+        handler.shutdown()
+
+
+@pytest_asyncio.fixture
+def metrics_controller(mock_logger):
+    """Create a real MetricsController instance."""
+    return MetricsController(logger=mock_logger)
+
+
+@pytest_asyncio.fixture
+def mock_minio_repository():
+    """Mock MinIO repository for object storage operations."""
+    mock_repo = MagicMock()
+    
+    # Mock repository methods
+    mock_repo.put_parquet_from_dataframe = AsyncMock(return_value='test-object-key')
+    mock_repo.get_parquet_as_dataframe = AsyncMock(return_value=pd.DataFrame())
+    mock_repo.minio_bucket = 'test-bucket'
+    
+    return mock_repo
+
+
+@pytest_asyncio.fixture
+def mock_pi_web_api_repository():
+    """Mock PI Web API repository for PI Web API operations."""
+    mock_repo = MagicMock()
+    mock_repo.write_value = AsyncMock(
+        return_value={
+            'Items': [
+                {
+                    'WebId': 'web_id_1'
+                }
+            ]
+        }
+    )
+    mock_repo.close = MagicMock()
+    return mock_repo
+
+@pytest_asyncio.fixture
+def mock_opc_repository():
+    """Mock OPC repository for OPC operations."""
+    mock_repo = MagicMock()
+    mock_repo.write_data = AsyncMock(
+        return_value=(True, {'response_time': 0.1})
+    )
+    mock_repo.disconnect = MagicMock()
+    return mock_repo
+
+@pytest_asyncio.fixture
+def patch_create_engine(postgres_engine):
+    """Patch create_engine to return test postgres_engine."""
+    with patch('sientia_do.temporal.activities.postgres.create_engine', return_value=postgres_engine):
+        yield
+
+
+@pytest_asyncio.fixture
+def patch_minio_repository(mock_minio_repository):
+    """Patch MinioRepository to return mock."""
+    with patch('laborious.utils.repository.minio_repository.MinioRepository', return_value=mock_minio_repository):
+        yield
+
+@pytest_asyncio.fixture
+def patch_pi_web_api_repository(mock_pi_web_api_repository):
+    """Patch MLflowRepository to return mock."""
+    with patch('laborious.activities.api.PIWebAPIClient', return_value=mock_pi_web_api_repository):
+        yield
+
+@pytest_asyncio.fixture
+def mock_mlflow_models():
+    """Create mock models for MLflow load_model methods."""
+    # Mock transform model - returns DataFrame with same index as input
+    mock_transform_model = MagicMock()
+    def mock_transform_predict(data):
+        num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
+        print(data.to_csv())
+        print(data.index)
+        result = pd.DataFrame({
+            'feature_1': [0.234] * num_rows,
+            'feature_2': [0.783] * num_rows,
+        })
+        result.index = data.index
+        return result
+    mock_transform_model.predict = MagicMock(side_effect=mock_transform_predict)
+    
+    # Mock predict model - returns array/list of predictions
+    mock_predict_model = MagicMock()
+    def mock_predict_predict(data):
+        num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
+        return [0.5] * num_rows
+    mock_predict_model.predict = MagicMock(side_effect=mock_predict_predict)
+    
+    # Mock PyFuncModel for compressed models
+    mock_pyfunc_model = MagicMock()
+    mock_pyfunc_model._model_impl = MagicMock()
+    mock_pyfunc_model._model_impl.python_model = mock_transform_model
+    
+    return {
+        'transform_model': mock_transform_model,
+        'predict_model': mock_predict_model,
+        'pyfunc_model': mock_pyfunc_model,
+    }
+
+
+@pytest_asyncio.fixture
+def patch_mlflow(mock_mlflow_models):
+    """Patch mlflow module in repository with load_model mocks."""
+    mock_mlflow = MagicMock()
+    
+    # Mock sklearn.load_model
+    def mock_sklearn_load_model(model_uri):
+        if 'data_model' in model_uri or 'transform' in model_uri.lower():
+            return mock_mlflow_models['transform_model']
+        return mock_mlflow_models['predict_model']
+    mock_mlflow.sklearn = MagicMock()
+    mock_mlflow.sklearn.load_model = MagicMock(side_effect=mock_sklearn_load_model)
+    
+    # Mock pyfunc.load_model
+    def mock_pyfunc_load_model(model_uri):
+        if 'artifacts' in model_uri or 'tmp' in model_uri:
+            return mock_mlflow_models['pyfunc_model']
+        if 'data_model' in model_uri or 'transform' in model_uri.lower():
+            return mock_mlflow_models['transform_model']
+        return mock_mlflow_models['predict_model']
+    mock_mlflow.pyfunc = MagicMock()
+    mock_mlflow.pyfunc.load_model = MagicMock(side_effect=mock_pyfunc_load_model)
+    
+    # Mock pytorch.load_model
+    mock_mlflow.pytorch = MagicMock()
+    mock_mlflow.pytorch.load_model = MagicMock(return_value=mock_mlflow_models['predict_model'])
+    
+    # Mock other mlflow methods that might be called
+    mock_mlflow.set_tracking_uri = MagicMock()
+    mock_mlflow.get_run = MagicMock(return_value=MagicMock(info=MagicMock(artifact_uri='mlflow-artifacts:/test_run_id')))
+    mock_mlflow.tracking = MagicMock()
+    mock_mlflow.tracking.MlflowClient = MagicMock(return_value=MagicMock(
+        search_registered_models=MagicMock(return_value=[MagicMock(name='test_model')]),
+        search_model_versions=MagicMock(return_value=[MagicMock(
+            current_stage='Production',
+            version='1',
+            source='runs:/artifacts/test_run_id'
+        )])
+    ))
+    
+    with patch('laborious.utils.repository.model_repository.mlflow', new=mock_mlflow):
+        yield mock_mlflow
+
+
+@pytest_asyncio.fixture(scope='function')
+async def test_activities(
+    postgres_engine,
+    postgres_container,
+    mock_logger,
+    notification_handler,
+    metrics_controller,
+    mock_minio_repository,
+    patch_create_engine,
+    patch_minio_repository,
+    patch_mlflow,
+    patch_pi_web_api_repository,
+    mock_opc_repository
+):
+    """
+    Create Activities instance with test dependencies.
+
+    This fixture creates a real Activities instance with:
+    - PostgreSQL database (via testcontainers)
+    - Mocked MinIO client
+    - Real NotificationHandler and MetricsController (with mocked underlying services)
+    """
+    activities = Activities(
+        postgres_config={
+            'host': 'localhost',
+            'port': postgres_container.get_exposed_port(5432),
+            'user': 'test',
+            'password': 'test',
+            'dbname': 'test',
+            'min_connections': 1,
+            'max_connections': 5,
+        },
+        mlflow_config={
+            'host': 'http://localhost',
+            'port': '5000',
+            'username': 'test',
+            'password': 'test',
+        },
+        minio_config={
+            'endpoint_url': 'http://localhost:9000',
+            'access_key': 'test',
+            'secret_key': 'test',
+            'region_name': 'us-east-1',
+            'default_bucket': 'test-bucket',
+        },
+        opc_config={},
+        pi_web_api_config={
+            'base_url': 'http://localhost:8080',
+            'auth_type': 'bearer',
+            'auth_token': 'test_token',
+        },
+        logger=mock_logger,
+        notification_handler=notification_handler,
+    )
+
+    activities.opc_repository = {
+        '1': mock_opc_repository,
+    }
+
+    try:
+        yield activities
+    finally:
+        # Cleanup - ALWAYS runs, even if test fails
+        await activities.shutdown()
+
+
+@pytest_asyncio.fixture(scope='function')
+async def temporal_test_env():
+    """Create Temporal test environment."""
+    env = await WorkflowEnvironment.start_time_skipping()
+    async with env:
+        yield env
+
+
+@pytest_asyncio.fixture(scope='function')
+async def temporal_worker(temporal_test_env, test_activities):
+    """Create Temporal worker with test activities."""
+    async with Worker(
+        temporal_test_env.client,
+        task_queue='test-queue',
+        workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
+        activities=[
+            test_activities.load_custom_query,
+            test_activities.get_last_timestamp,
+            test_activities.input_gate,
+            test_activities.request_transform,
+            test_activities.mlflow_response_gate,
+            test_activities.mlflow_content_gate,
+            test_activities.request_predict,
+            test_activities.repeat_last_prediction,
+            test_activities.format_prediction,
+            test_activities.format_transformed_data,
+            test_activities.format_default_prediction,
+            test_activities.write_pi_web_api_data,
+            test_activities.write_opc_data,
+            test_activities.export_data_to_postgres,
+            test_activities.write_metrics,
+        ],
+    ) as worker:
+        yield worker

e2e/scenarios.md

@@ -0,0 +1,509 @@
+# Test Scenarios for Predictions Batch Workflow
+
+This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.
+
+## Workflow Overview
+
+The `predictions_batch` workflow:
+1. Loads data using a custom SQL query
+2. Prepares prediction configuration
+3. Delegates to `prediction_process` child workflow which:
+   - Retrieves last timestamp for incremental processing
+   - Applies input data quality gates
+   - Executes MLFlow transform operation
+   - Validates transform response
+   - Executes MLFlow predict operation
+   - Validates predict response
+   - Delegates to `format_and_export_prediction` child workflow
+4. The `format_and_export_prediction` workflow:
+   - Formats prediction data (normal or default)
+   - Exports to PI Web API (optional)
+   - Exports to OPC server (optional)
+   - Exports to PostgreSQL
+   - Writes metrics
+
+---
+
+## 1. Predictions Batch - Main Workflow Scenarios
+
+### 1.1 Success Scenarios
+
+#### Scenario 1.1.1: Happy Path - Complete Success
+**Description**: Workflow completes successfully with valid SQL query and all activities succeed
+
+**Input**:
+- Valid `schedule_name`, `model_name`, `model_id`
+- Valid `query` returning non-empty DataFrame
+- Valid `schema`, `table_name`, `transform_table_name`
+- Optional `datetime_columns` for timestamp parsing
+- Optional `input_filters`, `mlflow_transform_filters`, `mlflow_predict_filters`
+- Optional `path_priority`, `opc_output_config`, `pi_web_api_output_config`
+
+**Expected Behavior**:
+- `load_custom_query` returns DataFrame with data
+- Workflow prepares prediction input with all configurations
+- `prediction_process` child workflow executes successfully
+- All gates pass with no issues
+- Transform and predict operations succeed
+- Data exported to PostgreSQL
+- Metrics written
+
+**Assertions**:
+- SQL query executed once
+- `prediction_process` workflow called with correct parameters
+- Data exists in PostgreSQL (predictions table)
+- Metrics recorded
+- No errors raised
+
+---
+
+### 1.2 Error Scenarios
+
+#### Scenario 1.2.1: SQL Query Execution Error
+**Description**: SQL query fails due to syntax error or connection issue
+
+**Input**:
+- Invalid SQL query (syntax error)
+- Or database connection unavailable
+
+**Expected Behavior**:
+- `load_custom_query` raises exception (caught by Temporal retry policy)
+- Notification sent with SQL error details
+- After retries, activity may return empty data or workflow may fail
+- If empty data returned, workflow completes with early exit via input gate
+
+**Assertions**:
+- Error notification sent
+- Workflow completes (either fails or exits early)
+- No data in predictions table
+
+---
+
+#### Scenario 1.2.2: Missing Required Parameters
+**Description**: Essential parameters missing from input
+
+**Input**:
+- Missing `query` or `model_id` or `schema` or `table_name`
+
+**Expected Behavior**:
+- Workflow or activity raises KeyError or validation error
+- Workflow fails immediately
+
+**Assertions**:
+- Workflow fails with parameter error
+- Error notification sent
+- No child workflow called
+
+---
+
+#### Scenario 1.2.3: Invalid Datetime Column Specification
+**Description**: Datetime column specified doesn't exist in query results
+
+**Input**:
+- `datetime_columns: ['nonexistent_column']`
+- Query results don't have this column
+
+**Expected Behavior**:
+- `load_custom_query` may raise KeyError or warning
+- Depending on implementation, workflow may fail or continue
+- Error notification sent
+
+**Assertions**:
+- Error raised or warning logged
+- Workflow behavior depends on error handling policy
+
+---
+
+## 2. Prediction Process - Child Workflow Scenarios
+
+### 2.1 Input gate Early Exit Scenarios
+
+#### Scenario 2.1.1: Input Gate Triggers CONTINUE
+**Description**: Input gate determines data should use previous prediction
+
+**Input**:
+- Data that should continue with input data as prediction
+- `input_filters` configured with `POLICY: 'CONTINUE'`
+- `path_priority` includes CONTINUE
+
+**Expected Behavior**:
+- `input_gate` returns `path_flag='CONTINUE'`
+- `path_flag_handler` calls export workflow with input data directly
+- MLFlow transform and predict skipped
+- Data exported as-is
+
+**Assertions**:
+- `input_gate` called
+- MLFlow operations NOT called
+- Export workflow called with original data
+- Workflow completes
+
+
+#### Scenario 2.1.2: Input Gate Triggers STOP
+**Description**: Input data quality gate fails with STOP policy
+
+**Input**:
+- Data with EMPTY_DATA or other critical issues
+- `input_filters` configured with `POLICY: 'STOP'`
+
+**Expected Behavior**:
+- `input_gate` returns `path_flag='STOP'`
+- `path_flag_handler` detects STOP
+- Workflow returns early without calling MLFlow
+- No prediction exported
+
+**Assertions**:
+- `input_gate` called
+- `path_flag_handler` returns True (early exit)
+- MLFlow transform NOT called
+- Export workflow NOT called
+- Workflow completes without error
+
+
+#### Scenario 2.1.3: Input Gate Triggers REPEAT
+**Description**: Input gate determines data should repeat last prediction
+
+**Input**:
+- Data with quality issues that require using previous prediction
+- `input_filters` configured with `POLICY: 'REPEAT'`
+- `path_priority` includes REPEAT
+
+**Expected Behavior**:
+- `input_gate` returns `path_flag='REPEAT'`
+- `path_flag_handler` calls `repeat_last_prediction` activity
+- MLFlow transform and predict skipped
+- Last prediction repeated and exported
+
+**Assertions**:
+- `input_gate` called
+- MLFlow operations NOT called
+- `repeat_last_prediction` activity called
+- Workflow completes
+
+---
+
+### 2.2 Transform gate Early Exit Scenarios
+
+#### Scenario 2.2.1: Transform Gate Triggers CONTINUE
+**Description**: Transform response gate determines data should continue despite issues
+
+**Input**:
+- Valid input data
+- Transform response has quality issues but policy is CONTINUE
+- `mlflow_transform_filters` configured with `POLICY: 'CONTINUE'`
+- `path_priority` includes CONTINUE
+
+**Expected Behavior**:
+- `request_transform` succeeds
+- `mlflow_response_gate` for transform returns `path_flag='CONTINUE'`
+- `path_flag_handler` calls export workflow with transform data
+- MLFlow predict skipped
+- Transform data exported as-is
+
+**Assertions**:
+- Transform completed
+- `mlflow_response_gate` called for transform
+- MLFlow predict NOT called
+- Export workflow called with transform data
+- Workflow completes
+
+---
+
+#### Scenario 2.2.2: Transform Gate Triggers STOP
+**Description**: Transform response validation fails with STOP policy
+
+**Input**:
+- Valid input data
+- Transform response has critical errors
+- `mlflow_transform_filters` configured with `POLICY: 'STOP'`
+
+**Expected Behavior**:
+- `request_transform` succeeds but response invalid
+- `mlflow_response_gate` for transform returns `path_flag='STOP'`
+- Workflow exits without calling predict or export
+
+**Assertions**:
+- Transform completed but validation failed
+- `mlflow_response_gate` called for transform
+- MLFlow predict NOT called
+- Export workflow NOT called
+- Workflow completes without error
+
+---
+
+#### Scenario 2.2.3: Transform Gate Triggers REPEAT
+**Description**: Transform response gate determines data should repeat last prediction
+
+**Input**:
+- Valid input data
+- Transform response has quality issues that require using previous prediction
+- `mlflow_transform_filters` configured with `POLICY: 'REPEAT'`
+- `path_priority` includes REPEAT
+
+**Expected Behavior**:
+- `request_transform` succeeds but response has issues
+- `mlflow_response_gate` for transform returns `path_flag='REPEAT'`
+- `path_flag_handler` calls `repeat_last_prediction` activity
+- MLFlow predict skipped
+- Last prediction repeated and exported
+
+**Assertions**:
+- Transform completed but validation triggered REPEAT
+- `mlflow_response_gate` called for transform
+- MLFlow predict NOT called
+- `repeat_last_prediction` activity called
+- Workflow completes
+
+---
+
+### 2.3 Predict gate Early Exit Scenarios
+
+#### Scenario 2.3.1: Predict Gate Triggers CONTINUE
+**Description**: Predict response gate determines data should continue despite issues
+
+**Input**:
+- Valid input and transform data
+- Predict response has quality issues but policy is CONTINUE
+- `mlflow_predict_filters` configured with `POLICY: 'CONTINUE'`
+- `path_priority` includes CONTINUE
+
+**Expected Behavior**:
+- `request_predict` succeeds
+- `mlflow_response_gate` for predict returns `path_flag='CONTINUE'`
+- `path_flag_handler` calls export workflow with predict data
+- Prediction exported despite quality issues
+
+**Assertions**:
+- Transform and predict completed
+- `mlflow_response_gate` called for predict
+- Export workflow called with predict data
+- Workflow completes
+
+---
+
+#### Scenario 2.3.2: Predict Gate Triggers STOP
+**Description**: Prediction validation fails with STOP policy
+
+**Input**:
+- Valid input and transform
+- Predict response has critical errors
+- `mlflow_predict_filters` configured with `POLICY: 'STOP'`
+
+**Expected Behavior**:
+- `request_predict` succeeds but response invalid
+- `mlflow_response_gate` for predict returns `path_flag='STOP'`
+- Workflow exits without export
+
+**Assertions**:
+- Transform completed
+- Predict completed but validation failed
+- Export workflow NOT called
+- Workflow completes without error
+
+---
+
+#### Scenario 2.3.3: Predict Gate Triggers REPEAT
+**Description**: Predict response gate determines data should repeat last prediction
+
+**Input**:
+- Valid input and transform data
+- Predict response has quality issues that require using previous prediction
+- `mlflow_predict_filters` configured with `POLICY: 'REPEAT'`
+- `path_priority` includes REPEAT
+
+**Expected Behavior**:
+- `request_predict` succeeds but response has issues
+- `mlflow_response_gate` for predict returns `path_flag='REPEAT'`
+- `path_flag_handler` calls `repeat_last_prediction` activity
+- Last prediction repeated and exported
+
+**Assertions**:
+- Transform and predict completed but validation triggered REPEAT
+- `mlflow_response_gate` called for predict
+- `repeat_last_prediction` activity called
+- Export workflow NOT called with current prediction
+- Workflow completes
+
+---
+
+## 3. Format and Export Prediction - Child Workflow Scenarios
+
+### 3.1 Success Scenarios
+
+#### Scenario 3.1.1: Default Prediction Export
+**Description**: Error prediction path creates default prediction
+
+**Input**:
+- `path_flag: 'ERROR'` or other non-None value (not STOP/CONTINUE/REPEAT)
+- `comment` provided with error details
+
+**Expected Behavior**:
+- `format_default_prediction` called instead of `format_prediction`
+- Default prediction created with error metadata
+- Exported to PostgreSQL only
+- Transformed data NOT processed
+- Metrics written
+
+**Assertions**:
+- `format_default_prediction` called
+- `format_prediction` NOT called
+- `format_transformed_data` NOT called
+- One PostgreSQL export only
+- Default values in prediction data
+- Comment included
+
+---
+
+#### Scenario 3.1.2: Export with OPC only
+**Description**: Export to PostgreSQL and OPC server only (no PI Web API)
+
+**Input**:
+- `path_flag: None`
+- `opc_output_config` configured with valid OPC settings
+- `pi_web_api_output_config: None` or `{}`
+
+**Expected Behavior**:
+- Normal formatting
+- PostgreSQL export executed
+- OPC export executed
+- PI Web API activity skipped
+- Metrics written with OPC metrics
+
+**Assertions**:
+- PI Web API activity NOT called
+- OPC activity called
+- PostgreSQL export called
+- Metrics written with `opc_metrics` populated
+
+---
+
+#### Scenario 3.1.3: Export with PI Web API only
+**Description**: Export to PostgreSQL and PI Web API only (no OPC)
+
+**Input**:
+- `path_flag: None`
+- `pi_web_api_output_config` configured with valid PI Web API settings
+- `opc_output_config: None` or `{}`
+
+**Expected Behavior**:
+- Normal formatting
+- PostgreSQL export executed
+- PI Web API export executed
+- OPC activity skipped
+- Metrics written without OPC metrics
+
+**Assertions**:
+- OPC activity NOT called
+- PI Web API activity called
+- PostgreSQL export called
+- Metrics written with empty `opc_metrics`
+
+---
+
+#### Scenario 3.1.4: Export Without Optional Outputs
+**Description**: Export only to PostgreSQL (no OPC or PI Web API)
+
+**Input**:
+- `path_flag: None`
+- `opc_output_config: None` or `{}`
+- `pi_web_api_output_config: None` or `{}`
+
+**Expected Behavior**:
+- Normal formatting
+- Only PostgreSQL export executed
+- OPC and PI Web API activities skipped
+- Metrics written without OPC metrics
+
+**Assertions**:
+- PI Web API activity NOT called
+- OPC activity NOT called
+- PostgreSQL export called
+- Metrics written with empty `opc_metrics`
+
+---
+
+#### Scenario 3.1.5: Export Without Transformed Data
+**Description**: Only prediction exported, no transform table
+
+**Input**:
+- `path_flag: None`
+- `transformed_data: None` or `save_transform: False`
+- `opc_output_config: None` or `{}`
+- `pi_web_api_output_config: None` or `{}`
+
+**Expected Behavior**:
+- Only prediction formatted and exported
+- Transform export skipped
+- Single PostgreSQL write
+
+**Assertions**:
+- `format_transformed_data` NOT called
+- One PostgreSQL export
+- Transform table remains empty
+
+---
+
+### 3.2 Error Scenarios
+
+#### Scenario 3.2.1: PI Web API Write Error
+**Description**: PI Web API export fails
+
+**Input**:
+- Valid prediction
+- PI Web API service unavailable or invalid config
+
+**Expected Behavior**:
+- `write_pi_web_api_data` raises exception
+- Notification sent
+- Workflow fails after retries
+- PostgreSQL export may not execute (depends on execution order)
+
+**Assertions**:
+- PI Web API error notification sent
+- Workflow fails
+- May impact subsequent exports
+
+---
+
+#### Scenario 3.2.2: OPC Write Error
+**Description**: OPC server write fails
+
+**Input**:
+- Valid prediction
+- OPC server unavailable or invalid configuration
+
+**Expected Behavior**:
+- `write_opc_data` raises exception
+- Notification sent
+- Workflow fails after retries
+
+**Assertions**:
+- OPC error notification sent
+- Workflow fails
+- PostgreSQL export may not execute
+
+---
+
+#### Scenario 3.2.3: PI Web API Partial Write Error
+**Description**: Two prediction tags attempt to be written to PI Web API, but only one succeeds
+
+**Input**:
+- Valid prediction
+- Two prediction tags configured
+- PI Web API returns partial success (one tag succeeds, one fails)
+
+**Expected Behavior**:
+- `write_pi_web_api_data` processes response
+- `process_pi_web_api_response` detects partial failure
+- Error confidence set (13)
+- Notification sent for failed tag
+- Workflow completes with error confidence
+
+**Assertions**:
+- One tag written successfully
+- One tag failed
+- Error confidence set in prediction
+- Error notification sent
+- Workflow completes
+
+---

e2e/test_predictions_batch_format_export.py

@@ -0,0 +1,829 @@
+"""
+End-to-end tests for PredictionsBatch workflow - Format and Export scenarios.
+"""
+
+import asyncio
+from datetime import datetime
+from unittest.mock import ANY, AsyncMock, patch, call
+
+import pandas as pd
+import pytest
+from sientia_do.notifications.models import NotificationLevel
+from sqlalchemy import text
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from laborious.activities.activities import Activities
+from laborious.workflows.predictions_batch import PredictionsBatch
+
+base_input_data = {
+    'schedule_name': 'test-schedule',
+    'model_name': 'test_model',
+    'model_id': 301,
+    'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 301',
+    'schema': 'predictions_schema',
+    'table_name': 'predictions',
+    'transform_table_name': 'transformed_data',
+    'input_filters': {
+        'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
+    },
+    'mlflow_transform_filters': {
+        'API_ERROR': {'policy': 'STOP', 'config': {}},
+    },
+    'mlflow_predict_filters': {
+        'API_ERROR': {'policy': 'STOP', 'config': {}},
+    },
+    'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+    'opc_output_config': {},
+    'pi_web_api_output_config': {},
+    'save_transform': True,
+    'prediction_store_policy': 'lts:1',
+    'model_config': {
+        'retention_minutes': 0,
+        'transform_flavor': 'sklearn',
+        'predict_flavor': 'sklearn',
+    },
+    'datetime_columns': ['timestamp', 'created_at'],
+}
+
+base_query = "SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = {model_id}"
+
+def get_base_input_data(model_id):
+    return {
+        **base_input_data,
+        'model_id': model_id,
+        'query': base_query.format(model_id=model_id),
+    }
+
+def insert_sample_data(postgres_engine, model_id, values: list):
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f"DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}"))
+        
+        values_sql = []
+        for i, value in enumerate(values):
+            values_sql.append(f"""
+                ({model_id}, 'sensor_{i+1}', {value}, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
+            """)
+
+        insert_sql = f"""
+            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            VALUES
+                {', '.join(values_sql)}
+        """
+        conn.execute(text(insert_sql))
+
+async def start_and_await_workflow(client, input_data, workflow_id):
+    handle = await client.start_workflow(
+        PredictionsBatch.run,
+        input_data,
+        id=workflow_id,
+        task_queue='test-queue',
+    )
+    print("[TEST] ✓ Workflow started")
+
+    print("\n[TEST] 3. Waiting for workflow completion...")
+    try:
+        await asyncio.wait_for(handle.result(), timeout=60.0)
+        print("[TEST] ✓ Workflow completed successfully")
+    except asyncio.TimeoutError:
+        pytest.fail("Workflow execution timed out after 60 seconds")
+
+def assert_prediction(
+    postgres_engine, model_id, prediction: float = 0.5,
+    prediction_confidence: int = 0, prediction_status: str = 'Good',
+    comments: str = '',
+):
+    """
+    Verify prediction was created with correct values in database
+    
+    Args:
+        postgres_engine: Database engine
+        model_id: Model ID to check
+        prediction: Expected prediction value (default 0.5 from mock)
+        prediction_confidence: Expected confidence value (default 0 for normal predictions)
+        prediction_status: Expected status (default 'Good')
+        comments: Expected comments (default empty string)
+    """
+    print("\n[TEST] 4. Verifying prediction was created with correct values...")
+    with postgres_engine.connect() as conn:
+        result_query = conn.execute(
+            text(f"SELECT model_id, prediction, prediction_confidence, prediction_status, comments FROM predictions_schema.predictions WHERE model_id = {model_id}")
+        )
+        prediction_rows = result_query.fetchall()
+        assert len(prediction_rows) == 1, f"Expected one prediction record, got {len(prediction_rows)}"
+
+        row = prediction_rows[0]
+        assert row[0] == model_id, f"Expected model_id={model_id}, got {row[0]}"
+        assert row[1] == prediction, f"Expected prediction={prediction}, got {row[1]}"
+        assert row[2] == prediction_confidence, f"Expected prediction_confidence={prediction_confidence}, got {row[2]}"
+        assert row[3] == prediction_status, f"Expected prediction_status='{prediction_status}', got {row[3]}"
+        assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_1_1_default_prediction_export(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.1.1: Default Prediction Export
+    
+    Description:
+    Error prediction path creates default prediction.
+    
+    Expected Behavior:
+    - format_default_prediction called instead of format_prediction
+    - Default prediction created with error metadata
+    - Exported to PostgreSQL only
+    - Transformed data NOT processed
+    - Metrics written
+    
+    Assertions:
+    - format_default_prediction called
+    - format_prediction NOT called
+    - format_transformed_data NOT called
+    - One PostgreSQL export only
+    - Default values in prediction data
+    - Comment included
+    """
+    client = temporal_test_env.client
+
+    model_id = 311
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {
+                'addr_1': {
+                    'data_type': 'float',
+                }
+            },
+            'confidence_tags': {
+                'addr_2': {
+                    'data_type': 'float',
+                }
+            },
+        }
+    }
+
+    print("\n[TEST] 2. Starting workflow that should create default prediction...")
+    workflow_id = f'test-default-prediction-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    test_activities.pi_web_api_client.write_value.assert_has_calls(
+        [
+            call(
+                web_ids=['web_id_1'],
+                value={
+                    'Timestamp': '2024-01-01 12:00:00+0000',
+                    'Value': 0.5,
+                },
+                endpoint='test_endpoint',
+                metadata={
+                    'model_id': 311,
+                    'model_name': 'test_model',
+                    'schedule_name': 'test-schedule',
+                    'workflow_name': 'predictions_batch',
+                },
+            ),
+            call(
+                web_ids=['web_id_2'],
+                value={
+                    'Timestamp': '2024-01-01 12:00:00+0000',
+                    'Value': 0,
+                },
+                endpoint='test_endpoint',
+                metadata={
+                    'model_id': 311,
+                    'model_name': 'test_model',
+                    'schedule_name': 'test-schedule',
+                    'workflow_name': 'predictions_batch',
+                },
+            ),
+        ],
+        any_order=True,
+    )
+
+    test_activities.opc_repository['1'].write_data.assert_has_calls(
+        [
+            call('addr_1', 0.5, 'float', ANY,
+            {
+                'model_id': 311,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }),
+            call('addr_2', 0, 'float', ANY,
+            {
+                'model_id': 311,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }),
+        ]
+    )
+
+    assert_prediction(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_1_2_export_with_opc_only(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.1.2: Export with OPC only
+    
+    Description:
+    Export to PostgreSQL and OPC server only (no PI Web API).
+    
+    Expected Behavior:
+    - Normal formatting
+    - PostgreSQL export executed
+    - OPC export executed
+    - PI Web API activity skipped
+    - Metrics written with OPC metrics
+    
+    Assertions:
+    - PI Web API activity NOT called
+    - OPC activity called
+    - PostgreSQL export called
+    - Metrics written with opc_metrics populated
+    """
+    client = temporal_test_env.client
+
+    model_id = 312
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {
+                'addr_1': {
+                    'data_type': 'float',
+                }
+            },
+            'confidence_tags': {
+                'addr_2': {
+                    'data_type': 'float',
+                }
+            },
+        }
+    }
+    input_data['pi_web_api_output_config'] = None  # No PI Web API config
+
+    print("\n[TEST] 2. Starting workflow with OPC only...")
+    workflow_id = f'test-opc-only-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    test_activities.opc_repository['1'].write_data.assert_has_calls(
+        [
+            call('addr_1', 0.5, 'float', ANY,
+            {
+                'model_id': 312,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }),
+            call('addr_2', 0, 'float', ANY,
+            {
+                'model_id': 312,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }),
+        ]
+    )
+
+    test_activities.pi_web_api_client.write_value.assert_not_called()
+
+    assert_prediction(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_1_3_export_with_pi_web_api_only(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.1.3: Export with PI Web API only
+    
+    Description:
+    Export to PostgreSQL and PI Web API only (no OPC).
+    
+    Expected Behavior:
+    - Normal formatting
+    - PostgreSQL export executed
+    - PI Web API export executed
+    - OPC activity skipped
+    - Metrics written without OPC metrics
+    
+    Assertions:
+    - OPC activity NOT called
+    - PI Web API activity called
+    - PostgreSQL export called
+    - Metrics written with empty opc_metrics
+    """
+    client = temporal_test_env.client
+
+    model_id = 313
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+    input_data['opc_output_config'] = None  # No OPC config
+
+    print("\n[TEST] 2. Starting workflow with PI Web API only...")
+    workflow_id = f'test-pi-api-only-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    test_activities.pi_web_api_client.write_value.assert_has_calls(
+        [
+            call(
+                web_ids=['web_id_1'],
+                value={
+                    'Timestamp': '2024-01-01 12:00:00+0000',
+                    'Value': 0.5,
+                },
+                endpoint='test_endpoint',
+                metadata={
+                    'model_id': 313,
+                    'model_name': 'test_model',
+                    'schedule_name': 'test-schedule',
+                    'workflow_name': 'predictions_batch',
+                },
+            ),
+            call(
+                web_ids=['web_id_2'],
+                value={
+                    'Timestamp': '2024-01-01 12:00:00+0000',
+                    'Value': 0,
+                },
+                endpoint='test_endpoint',
+                metadata={
+                    'model_id': 313,
+                    'model_name': 'test_model',
+                    'schedule_name': 'test-schedule',
+                    'workflow_name': 'predictions_batch',
+                },
+            ),
+        ],
+        any_order=True,
+    )
+
+    test_activities.opc_repository['1'].write_data.assert_not_called()
+
+    assert_prediction(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_1_4_export_without_optional_outputs(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.1.4: Export Without Optional Outputs
+    
+    Description:
+    Export only to PostgreSQL (no OPC or PI Web API).
+    
+    Expected Behavior:
+    - Normal formatting
+    - Only PostgreSQL export executed
+    - OPC and PI Web API activities skipped
+    - Metrics written without OPC metrics
+    
+    Assertions:
+    - PI Web API activity NOT called
+    - OPC activity NOT called
+    - PostgreSQL export called
+    - Metrics written with empty opc_metrics
+    """
+    client = temporal_test_env.client
+
+    model_id = 314
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = None  # No OPC config
+    input_data['pi_web_api_output_config'] = None  # No PI Web API config
+
+    print("\n[TEST] 2. Starting workflow without optional outputs...")
+    workflow_id = f'test-no-optional-outputs-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    test_activities.pi_web_api_client.write_value.assert_not_called()
+    test_activities.opc_repository['1'].write_data.assert_not_called()
+
+    assert_prediction(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_1_5_export_without_transformed_data(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.1.5: Export Without Transformed Data
+    
+    Description:
+    Only prediction exported, no transform table.
+    
+    Expected Behavior:
+    - Only prediction formatted and exported
+    - Transform export skipped
+    - Single PostgreSQL write
+    
+    Assertions:
+    - format_transformed_data NOT called
+    - One PostgreSQL export
+    - Transform table remains empty
+    """
+    client = temporal_test_env.client
+
+    model_id = 315
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f"DELETE FROM predictions_schema.transformed_data WHERE model_id = {model_id}"))
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['save_transform'] = False  # Don't save transformed data
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {
+                'addr_1': {
+                    'data_type': 'float',
+                }
+            },
+            'confidence_tags': {
+                'addr_2': {
+                    'data_type': 'float',
+                }
+            },
+        }
+    }
+
+    print("\n[TEST] 2. Starting workflow without transformed data export...")
+    workflow_id = f'test-no-transform-export-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    test_activities.pi_web_api_client.write_value.assert_has_calls(
+        [
+            call(
+                web_ids=['web_id_1'],
+                value={
+                    'Timestamp': '2024-01-01 12:00:00+0000',
+                    'Value': 0.5,
+                },
+                endpoint='test_endpoint',
+                metadata={
+                    'model_id': 315,
+                    'model_name': 'test_model',
+                    'schedule_name': 'test-schedule',
+                    'workflow_name': 'predictions_batch',
+                },
+            ),
+            call(
+                web_ids=['web_id_2'],
+                value={
+                    'Timestamp': '2024-01-01 12:00:00+0000',
+                    'Value': 0,
+                },
+                endpoint='test_endpoint',
+                metadata={
+                    'model_id': 315,
+                    'model_name': 'test_model',
+                    'schedule_name': 'test-schedule',
+                    'workflow_name': 'predictions_batch',
+                },
+            ),
+        ],
+        any_order=True,
+    )
+
+    test_activities.opc_repository['1'].write_data.assert_has_calls(
+        [
+            call('addr_1', 0.5, 'float', ANY,
+            {
+                'model_id': 315,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }),
+            call('addr_2', 0, 'float', ANY,
+            {
+                'model_id': 315,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }),
+        ]
+    )
+
+    with postgres_engine.connect() as conn:
+        result_query = conn.execute(
+            text(f"SELECT COUNT(*) FROM predictions_schema.transformed_data WHERE model_id = {model_id}")
+        )
+        count = result_query.scalar()
+        assert count == 0, f"Expected transform table to be empty, but found {count} records"
+
+    assert_prediction(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_2_1_pi_web_api_write_error(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.1: PI Web API Write Error
+    
+    Description:
+    PI Web API export fails.
+    
+    Expected Behavior:
+    - write_pi_web_api_data raises exception
+    - Notification sent
+    - Workflow fails after retries
+    - PostgreSQL export may not execute (depends on execution order)
+    
+    Assertions:
+    - PI Web API error notification sent
+    - Workflow fails
+    - May impact subsequent exports
+    """
+    client = temporal_test_env.client
+
+    model_id = 321
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    test_activities.pi_web_api_client.write_value.side_effect = Exception(
+        "PI Web API service unavailable")
+
+    input_data = get_base_input_data(model_id)
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {
+                'addr_1': {
+                    'data_type': 'float',
+                }
+            },
+            'confidence_tags': {
+                'addr_2': {
+                    'data_type': 'float',
+                }
+            },
+        }
+    }
+
+    print("\n[TEST] 2. Starting workflow that should fail on PI Web API write...")
+    workflow_id = f'test-pi-api-error-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    assert_prediction(
+        postgres_engine, model_id,
+        prediction_confidence=13,
+        comments='PI Web API service unavailable',
+    )
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_2_2_opc_write_error(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.2: OPC Write Error
+    
+    Description:
+    OPC server write fails.
+    
+    Expected Behavior:
+    - write_opc_data raises exception
+    - Notification sent
+    - Workflow fails after retries
+    
+    Assertions:
+    - OPC error notification sent
+    - Workflow fails
+    - PostgreSQL export may not execute
+    """
+    client = temporal_test_env.client
+
+    model_id = 322
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    test_activities.opc_repository['1'].write_data.return_value = (False, {
+        'notification_id': 'OPC_WRITE_DATA_ERROR_1',
+        'message': 'OPC server unavailable',
+        'block': 'opc_repository',
+        'level': NotificationLevel.ERROR,
+        'attachment_content': 'OPC server unavailable',
+    })
+    
+    input_data = get_base_input_data(model_id)
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {
+                'addr_1': {
+                    'data_type': 'float',
+                }
+            },
+            'confidence_tags': {
+                'addr_2': {
+                    'data_type': 'float',
+                }
+            },
+        }
+    }
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+
+    print("\n[TEST] 2. Starting workflow that should fail on OPC write...")
+    workflow_id = f'test-opc-error-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    assert_prediction(
+        postgres_engine, model_id,
+        prediction_confidence=12,
+        comments='Some data could not be written to OPC servers',
+    )
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_3_2_3_pi_web_api_partial_write_error(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 3.2.3: PI Web API Partial Write Error
+    
+    Description:
+    Two prediction tags attempt to be written to PI Web API, but only one succeeds.
+    
+    Expected Behavior:
+    - write_pi_web_api_data processes response
+    - process_pi_web_api_response detects partial failure
+    - Error confidence set (13)
+    - Notification sent for failed tag
+    - Workflow completes with error confidence
+    
+    Assertions:
+    - One tag written successfully
+    - One tag failed
+    - Error confidence set in prediction
+    - Error notification sent
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 323
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    test_activities.pi_web_api_client.write_value = AsyncMock(side_effect=[
+        {
+            'Items': [
+                {
+                    'WebId': 'web_id_1',
+                    'Errors': [],
+                },
+            ]
+        },
+        Exception('Tag write failed'),
+        {
+            'Items': [
+                {
+                    'WebId': 'web_id_2',
+                    'Errors': [],
+                },
+            ]
+        },
+    ])
+
+    input_data = get_base_input_data(model_id)
+    input_data['pi_web_api_output_config'] = {
+        'endpoint': 'test_endpoint',
+        'prediction_tags': {'tag_1': 'web_id_1', 'tag_3': 'web_id_3'},
+        'confidence_tags': {'tag_2': 'web_id_2'},
+    }
+    input_data['opc_output_config'] = {
+        '1': {
+            'prediction_tags': {
+                'addr_1': {
+                    'data_type': 'float',
+                }
+            },
+            'confidence_tags': {
+                'addr_2': {
+                    'data_type': 'float',
+                }
+            },
+        }
+    }
+
+    print("\n[TEST] 2. Starting workflow with partial PI Web API write error...")
+    workflow_id = f'test-pi-api-partial-error-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    assert_prediction(
+        postgres_engine, model_id,
+        prediction_confidence=13,
+        comments="The number of written tags does not match the number of tag names: Expected ['tag_1', 'tag_3'] tags, but ['tag_1'] tags were written.",
+    )
+    
+    print("\n[TEST] ✓ All assertions passed!")

e2e/test_predictions_batch_main_workflow.py

@@ -0,0 +1,450 @@
+"""
+End-to-end tests for PredictionsBatch workflow - Main workflow scenarios.
+"""
+
+import asyncio
+from datetime import datetime
+
+import pandas as pd
+import pytest
+from sqlalchemy import text
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from laborious.activities.activities import Activities
+from laborious.workflows.predictions_batch import PredictionsBatch
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_1_1_1_happy_path_complete_success(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 1.1.1: Happy Path - Complete Success
+    
+    Description:
+    Workflow completes successfully with valid SQL query and all activities succeed.
+    
+    Process Flow:
+    1. load_custom_query returns DataFrame with sensor data
+    2. Workflow prepares prediction input with all configurations
+    3. prediction_process child workflow executes:
+       - get_last_timestamp retrieves last processing timestamp
+       - input_gate validates data quality (passes)
+       - request_transform calls MLFlow transform (mocked, returns features)
+       - mlflow_response_gate validates transform response (passes)
+       - mlflow_content_gate validates transform content (passes)
+       - request_predict calls MLFlow predict (mocked, returns predictions)
+       - mlflow_response_gate validates predict response (passes)
+       - mlflow_content_gate validates predict content (passes)
+    4. format_and_export_prediction child workflow executes:
+       - format_prediction formats the prediction data
+       - format_transformed_data formats transformed data (if save_transform=True)
+       - export_data_to_postgres saves to database
+       - write_metrics records execution metrics
+    
+    Expected Behavior:
+    - All activities execute successfully without errors
+    - All gates pass with no quality issues
+    - Transform and predict operations succeed (mocked)
+    - Data exported to PostgreSQL predictions table
+    - Transformed data exported to transformed_data table (if save_transform=True)
+    - Metrics written successfully
+    
+    Assertions:
+    - Workflow completes without raising exceptions
+    - Data exists in PostgreSQL predictions table with correct model_id
+    - Data exists in transformed_data table (if save_transform=True)
+    - Prediction data has expected structure (jsonb with predictions)
+    - All required fields are populated (model_id, model_name, timestamp, etc)
+    """
+    client = temporal_test_env.client
+
+    print("\n[TEST] 1. Inserting test data into PostgreSQL...")
+    # Insert test data directly into PostgreSQL
+    # The load_custom_query activity will fetch this data with a real SQL query
+    with postgres_engine.begin() as conn:
+        # Clear any existing data for this model_id
+        conn.execute(text("DELETE FROM predictions_schema.laborious_data WHERE model_id = 123"))
+        
+        # Insert sensor data that the workflow will query
+        insert_sql = """
+            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            VALUES 
+                (123, 'sensor_1', 23.5, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00'),
+                (123, 'sensor_2', 78.2, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00'),
+                (123, 'sensor_3', 120.8, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
+        """
+        conn.execute(text(insert_sql))
+    print("[TEST] ✓ Data inserted successfully")
+
+    # Prepare input data for PredictionsBatch workflow
+    input_data = {
+        'metadata': {
+            'metadata': {
+                'model_id': 123,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }
+        },
+        'schedule_name': 'test-schedule',
+        'model_name': 'test_model',
+        'model_id': 123,
+        'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 123',
+        'schema': 'predictions_schema',
+        'table_name': 'predictions',
+        'transform_table_name': 'transformed_data',
+        'input_filters': {
+            'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_transform_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_predict_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+        'opc_output_config': {},
+        'pi_web_api_output_config': {},
+        'save_transform': True,
+        'prediction_store_policy': 'lts:1',
+        'model_config': {
+            'retention_minutes': 0,
+            'transform_flavor': 'sklearn',
+            'predict_flavor': 'sklearn',
+        },
+        'datetime_columns': ['timestamp', 'created_at'],
+    }
+
+    # Start workflow
+    print("\n[TEST] 2. Starting workflow...")
+    workflow_id = f'test-predictions-batch-{datetime.now().timestamp()}'
+    print(f"[TEST] Workflow ID: {workflow_id}")
+    
+    handle = await client.start_workflow(
+        PredictionsBatch.run,
+        input_data,
+        id=workflow_id,
+        task_queue='test-queue',
+    )
+    print("[TEST] ✓ Workflow started")
+
+    # Wait for workflow completion with timeout
+    print("\n[TEST] 3. Waiting for workflow completion (timeout: 60s)...")
+    try:
+        await asyncio.wait_for(handle.result(), timeout=60.0)  # 60 seconds timeout
+        print("[TEST] ✓ Workflow completed successfully")
+    except asyncio.TimeoutError:
+        print("[TEST] ✗ Workflow TIMEOUT after 60 seconds!")
+        pytest.fail("Workflow execution timed out after 60 seconds")
+
+    # Verify data was stored in PostgreSQL - use single connection
+    schema_name = 'predictions_schema'
+    predictions_table = 'predictions'
+    transformed_table = 'transformed_data'
+    full_predictions_table = f"{schema_name}.{predictions_table}"
+    full_transformed_table = f"{schema_name}.{transformed_table}"
+
+    # Use a single connection for all verification queries
+    print("\n[TEST] 4. Verifying results in PostgreSQL...")
+    with postgres_engine.connect() as conn:
+        # Verify prediction data
+        result_query = conn.execute(
+            text(f"SELECT model_id, prediction, prediction_confidence, response_time, prediction_status, comments FROM {full_predictions_table} WHERE model_id = 123")
+        )
+        prediction_rows = result_query.fetchall()
+
+        print(f"[TEST] Found {len(prediction_rows)} prediction record(s)")
+        assert len(prediction_rows) == 1, "Expected one prediction record"
+        
+        # Verify first row has expected structure
+        row = prediction_rows[0]
+        print(f"[TEST] Prediction: {row}")
+        assert row[0] == 123, f"Expected model_id=123, got {row[0]}"
+        assert row[1] == 0.5, f"Expected prediction=0.5, got {row[1]}"
+        assert row[2] == 0, f"Expected prediction_confidence=0.9, got {row[2]}"
+        assert row[3] is not None, f"Expected response_time=0.1, got {row[3]}"
+        assert row[4] == 'Good', f"Expected prediction_status='Good', got {row[4]}"
+        assert row[5] == '', f"Expected comments='', got {row[5]}"
+        print("[TEST] ✓ Prediction data verified")
+
+        # Verify transformed data
+        result_query = conn.execute(
+            text(f"SELECT model_id, variable, value FROM {full_transformed_table} WHERE model_id = 123")
+        )
+        transformed_rows = result_query.fetchall()
+        print(f"[TEST] Found {len(transformed_rows)} transformed data record(s)")
+        assert len(transformed_rows) == 2, "Expected two transformed data records"
+        row_1 = transformed_rows[0]
+        print(f"[TEST] Transformed data: {row_1}")
+        assert row_1[0] == 123, f"Expected model_id=123, got {row_1[0]}"
+        assert row_1[1] == 'feature_1', f"Expected variable='sensor_1', got {row_1[1]}"
+        assert float(row_1[2]) == 0.234, f"Expected value=0.234, got {row_1[2]}"
+        row_2 = transformed_rows[1]
+        print(f"[TEST] Transformed data: {row_2}")
+        assert row_2[0] == 123, f"Expected model_id=123, got {row_2[0]}"
+        assert row_2[1] == 'feature_2', f"Expected variable='sensor_2', got {row_2[1]}"
+        assert float(row_2[2]) == 0.783, f"Expected value=0.783, got {row_2[2]}"
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_1_2_1_sql_query_execution_error(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 1.2.1: SQL Query Execution Error
+    
+    Description:
+    SQL query fails due to syntax error or connection issue.
+    
+    Expected Behavior:
+    - load_custom_query raises exception (caught by Temporal retry policy)
+    - Notification sent with SQL error details
+    - After retries, activity may return empty data or workflow may fail
+    - If empty data returned, workflow completes with early exit via input gate
+    
+    Assertions:
+    - Error notification sent
+    - Workflow completes (either fails or exits early)
+    - No data in predictions table
+    """
+    client = temporal_test_env.client
+
+    input_data = {
+        'metadata': {
+            'metadata': {
+                'model_id': 128,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }
+        },
+        'schedule_name': 'test-schedule',
+        'model_name': 'test_model',
+        'model_id': 128,
+        'query': 'SELECT * FROM nonexistent_table WHERE invalid_syntax =',  # Invalid SQL
+        'schema': 'predictions_schema',
+        'table_name': 'predictions',
+        'transform_table_name': 'transformed_data',
+        'input_filters': {
+            'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_transform_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_predict_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+        'opc_output_config': {},
+        'pi_web_api_output_config': {},
+        'save_transform': True,
+        'prediction_store_policy': 'lts:1',
+        'model_config': {
+            'retention_minutes': 0,
+            'transform_flavor': 'sklearn',
+            'predict_flavor': 'sklearn',
+        },
+    }
+
+    print("\n[TEST] 1. Starting workflow with invalid SQL query...")
+    workflow_id = f'test-sql-error-{datetime.now().timestamp()}'
+    
+    handle = await client.start_workflow(
+        PredictionsBatch.run,
+        input_data,
+        id=workflow_id,
+        task_queue='test-queue',
+    )
+    print("[TEST] ✓ Workflow started")
+
+    print("\n[TEST] 2. Waiting for workflow completion...")
+    try:
+        await asyncio.wait_for(handle.result(), timeout=60.0)
+        print("[TEST] ✓ Workflow completed (may have exited early due to empty data)")
+    except Exception as e:
+        print(f"[TEST] ✓ Workflow failed as expected: {type(e).__name__}")
+    
+    # Verify no predictions were created (regardless of whether workflow failed or exited early)
+    print("\n[TEST] 3. Verifying no predictions were created...")
+    with postgres_engine.connect() as conn:
+        result_query = conn.execute(
+            text("SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = 128")
+        )
+        count = result_query.scalar()
+        assert count == 0, f"Expected no predictions, but found {count} records"
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_1_2_2_missing_required_parameters(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 1.2.2: Missing Required Parameters
+    
+    Description:
+    Essential parameters missing from input.
+    
+    Expected Behavior:
+    - Workflow or activity raises KeyError or validation error
+    - Workflow fails immediately
+    
+    Assertions:
+    - Workflow fails with parameter error
+    - Error notification sent
+    - No child workflow called
+    """
+    client = temporal_test_env.client
+
+    # Missing 'query' parameter
+    input_data = {
+        'metadata': {
+            'metadata': {
+                'model_id': 129,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }
+        },
+        'schedule_name': 'test-schedule',
+        'model_name': 'test_model',
+        'model_id': 129,
+        # 'query' is missing
+        'schema': 'predictions_schema',
+        'table_name': 'predictions',
+        'transform_table_name': 'transformed_data',
+    }
+
+    print("\n[TEST] 1. Starting workflow with missing required parameter...")
+    workflow_id = f'test-missing-param-{datetime.now().timestamp()}'
+    
+    handle = await client.start_workflow(
+        PredictionsBatch.run,
+        input_data,
+        id=workflow_id,
+        task_queue='test-queue',
+    )
+    print("[TEST] ✓ Workflow started")
+
+    print("\n[TEST] 2. Waiting for workflow to fail...")
+    try:
+        await asyncio.wait_for(handle.result(), timeout=60.0)
+        pytest.fail("Expected workflow to fail, but it completed successfully")
+    except Exception as e:
+        print(f"[TEST] ✓ Workflow failed as expected: {type(e).__name__}")
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_1_2_3_invalid_datetime_column_specification(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 1.2.3: Invalid Datetime Column Specification
+    
+    Description:
+    Datetime column specified doesn't exist in query results.
+    
+    Expected Behavior:
+    - load_custom_query may raise KeyError or warning
+    - Depending on implementation, workflow may fail or continue
+    - Error notification sent
+    
+    Assertions:
+    - Error raised or warning logged
+    - Workflow behavior depends on error handling policy
+    """
+    client = temporal_test_env.client
+
+    print("\n[TEST] 1. Inserting test data...")
+    with postgres_engine.begin() as conn:
+        conn.execute(text("DELETE FROM predictions_schema.laborious_data WHERE model_id = 130"))
+        
+        insert_sql = """
+            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            VALUES 
+                (130, 'sensor_1', 23.5, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
+        """
+        conn.execute(text(insert_sql))
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = {
+        'metadata': {
+            'metadata': {
+                'model_id': 130,
+                'model_name': 'test_model',
+                'schedule_name': 'test-schedule',
+                'workflow_name': 'predictions_batch',
+            }
+        },
+        'schedule_name': 'test-schedule',
+        'model_name': 'test_model',
+        'model_id': 130,
+        'query': 'SELECT timestamp, variable, value FROM predictions_schema.laborious_data WHERE model_id = 130',
+        'schema': 'predictions_schema',
+        'table_name': 'predictions',
+        'transform_table_name': 'transformed_data',
+        'input_filters': {
+            'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_transform_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_predict_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+        'opc_output_config': {},
+        'pi_web_api_output_config': {},
+        'save_transform': True,
+        'prediction_store_policy': 'lts:1',
+        'model_config': {
+            'retention_minutes': 0,
+            'transform_flavor': 'sklearn',
+            'predict_flavor': 'sklearn',
+        },
+        'datetime_columns': ['nonexistent_column'],  # Column doesn't exist in query result
+    }
+
+    print("\n[TEST] 2. Starting workflow with invalid datetime column...")
+    workflow_id = f'test-invalid-datetime-col-{datetime.now().timestamp()}'
+    
+    handle = await client.start_workflow(
+        PredictionsBatch.run,
+        input_data,
+        id=workflow_id,
+        task_queue='test-queue',
+    )
+    print("[TEST] ✓ Workflow started")
+
+    print("\n[TEST] 3. Waiting for workflow completion or failure...")
+    try:
+        await asyncio.wait_for(handle.result(), timeout=60.0)
+        # Workflow may complete or fail depending on error handling
+        print("[TEST] ✓ Workflow completed (may have handled error gracefully)")
+    except Exception as e:
+        print(f"[TEST] ✓ Workflow failed as expected: {type(e).__name__}")
+    
+    print("\n[TEST] ✓ Test completed!")

e2e/test_predictions_batch_prediction_process.py

@@ -0,0 +1,629 @@
+"""
+End-to-end tests for PredictionsBatch workflow - Prediction Process scenarios.
+"""
+
+import asyncio
+from datetime import datetime
+from decimal import Decimal
+from unittest.mock import MagicMock, patch
+
+import pandas as pd
+import pytest
+from pytz import timezone
+from sqlalchemy import text
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import Worker
+
+from laborious.activities.activities import Activities
+from laborious.workflows.predictions_batch import PredictionsBatch
+
+base_input_data = {
+        'schedule_name': 'test-schedule',
+        'model_name': 'test_model',
+        'model_id': 201,
+        'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 201',
+        'schema': 'predictions_schema',
+        'table_name': 'predictions',
+        'transform_table_name': 'transformed_data',
+        'input_filters': {
+            'SPECIFIC_VARIABLES_NULL_VALUES': {
+                'policy': 'CONTINUE',  # Continue despite issues, not STOP
+                'config': {'variables': ['sensor_1']},
+            },
+        },
+        'mlflow_transform_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'mlflow_predict_filters': {
+            'API_ERROR': {'policy': 'STOP', 'config': {}},
+        },
+        'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
+        'opc_output_config': {},
+        'pi_web_api_output_config': {},
+        'save_transform': True,
+        'prediction_store_policy': 'lts:1',
+        'model_config': {
+            'retention_minutes': 0,
+            'transform_flavor': 'sklearn',
+            'predict_flavor': 'sklearn',
+        },
+        'datetime_columns': ['timestamp', 'created_at'],
+    }
+
+base_query = "SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = {model_id}"
+
+def get_base_input_data(model_id):
+    return {
+        **base_input_data,
+        'model_id': model_id,
+        'query': base_query.format(model_id=model_id),
+    }
+
+def insert_sample_data(postgres_engine, model_id, values: list[tuple]):
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f"DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}"))
+        
+        # Insert data with some null values (quality issue)
+
+        values_sql = []
+        for i, value in enumerate(values):
+            values_sql.append(f"""
+                ({model_id}, 'sensor_{i+1}', {value}, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
+            """)
+
+        insert_sql = f"""
+            INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
+            VALUES
+                {', '.join(values_sql)}
+        """
+        conn.execute(text(insert_sql))
+
+
+def insert_sample_prediction(postgres_engine, model_id):
+    with postgres_engine.begin() as conn:
+        conn.execute(text(f"DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}"))
+        
+        # Insert data with some null values (quality issue)
+
+        insert_sql = f"""
+            INSERT INTO predictions_schema.predictions (model_id, timestamp, prediction, prediction_confidence, prediction_status, comments, response_time)
+            VALUES
+               ({model_id}, '2024-01-01 12:00:00+00:00', 10, 0, 'Good', '', 0.1)
+        """
+        conn.execute(text(insert_sql))
+
+        return (model_id, Decimal(10), Decimal(0), 'Good')
+
+async def start_and_await_workflow(client, input_data, workflow_id):
+    handle = await client.start_workflow(
+        PredictionsBatch.run,
+        input_data,
+        id=workflow_id,
+        task_queue='test-queue',
+    )
+    print("[TEST] ✓ Workflow started")
+
+    print("\n[TEST] 3. Waiting for workflow completion...")
+    try:
+        await asyncio.wait_for(handle.result(), timeout=60.0)
+        print("[TEST] ✓ Workflow completed successfully")
+    except asyncio.TimeoutError:
+        pytest.fail("Workflow execution timed out after 60 seconds")
+
+def assert_continue(
+    postgres_engine, model_id, prediction_confidence: Decimal = 2,
+    comments: str = 'Input data with bad quality',
+):
+    print("\n[TEST] 4. Verifying prediction was created despite warnings...")
+    with postgres_engine.connect() as conn:
+        result_query = conn.execute(
+            text(f"SELECT model_id, prediction, prediction_confidence, prediction_status, comments FROM predictions_schema.predictions WHERE model_id = {model_id}")
+        )
+        prediction_rows = result_query.fetchall()
+        assert len(prediction_rows) == 1, "Expected one prediction record despite warnings"
+
+        # Assert prediction value is 0 and other fields
+        row = prediction_rows[0]
+        assert row[1] == 0, f"Expected prediction=0, got {row[1]}"
+        assert row[2] == prediction_confidence, f"Expected prediction_confidence={prediction_confidence}, got {row[2]}"
+        assert row[3] == 'Bad', f"Expected prediction_status='Bad', got {row[3]}"
+        assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
+
+
+def assert_stop(postgres_engine, model_id):
+    print("\n[TEST] 4. Verifying no predictions were created...")
+    with postgres_engine.connect() as conn:
+        result_query = conn.execute(
+            text(f"SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = {model_id}")
+        )
+        count = result_query.scalar()
+        assert count == 0, f"Expected no predictions, but found {count} records"
+
+def assert_repeat(postgres_engine, model_id, last_prediction: list):
+    print("\n[TEST] 4. Verifying prediction was repeated...")
+    with postgres_engine.connect() as conn:
+        result_query = conn.execute(
+            text(f'SELECT model_id, prediction, prediction_confidence, prediction_status FROM predictions_schema.predictions WHERE model_id = {model_id}')
+        )
+        prediction_rows = result_query.fetchall()
+        print(prediction_rows)
+        assert len(prediction_rows) == 2, "Expected two prediction records"
+        assert prediction_rows[0] == last_prediction, f"Expected first prediction to be the same as the last prediction, got {prediction_rows[0]}, expected {last_prediction}"
+        assert prediction_rows[1] == last_prediction, f"Expected second prediction to be the same as the last prediction, got {prediction_rows[1]}, expected {last_prediction}"
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_1_1_input_gate_triggers_continue(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 2.1.1: Input Gate Triggers CONTINUE
+    
+    Description:
+    Input gate determines data should use previous prediction.
+    
+    Expected Behavior:
+    - input_gate returns path_flag='CONTINUE'
+    - path_flag_handler calls export workflow with input data directly
+    - MLFlow transform and predict skipped
+    - Data exported as-is
+    
+    Assertions:
+    - input_gate called
+    - MLFlow operations NOT called
+    - Export workflow called with original data
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 211
+
+    print("\n[TEST] 1. Inserting test data...")
+
+    insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
+
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+
+
+    print("\n[TEST] 2. Starting workflow with CONTINUE policy...")
+    workflow_id = f'test-continue-policy-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_continue(postgres_engine, model_id)
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_1_2_input_gate_triggers_stop(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 2.1.2: Input Gate Triggers STOP
+    
+    Description:
+    Input data quality gate fails with STOP policy.
+    
+    Expected Behavior:
+    - input_gate returns path_flag='STOP'
+    - path_flag_handler detects STOP
+    - Workflow returns early without calling MLFlow
+    - No prediction exported
+    
+    Assertions:
+    - input_gate called
+    - path_flag_handler returns True (early exit)
+    - MLFlow transform NOT called
+    - Export workflow NOT called
+    - Workflow completes without error
+    """
+    client = temporal_test_env.client
+
+    model_id = 212
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['policy'] = 'STOP'
+
+    print("\n[TEST] 2. Starting workflow that should stop at input gate...")
+    workflow_id = f'test-input-stop-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+
+    assert_stop(postgres_engine, model_id)
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_1_3_input_gate_triggers_repeat(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+):
+    """
+    Scenario 2.1.3: Input Gate Triggers REPEAT
+    
+    Description:
+    Input gate determines data should repeat last prediction.
+    
+    Expected Behavior:
+    - input_gate returns path_flag='REPEAT'
+    - path_flag_handler calls repeat_last_prediction activity
+    - MLFlow transform and predict skipped
+    - Last prediction repeated and exported
+    
+    Assertions:
+    - input_gate called
+    - MLFlow operations NOT called
+    - repeat_last_prediction activity called
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 213
+
+    print("\n[TEST] 1. Inserting test data and previous prediction...")
+    insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
+    data = insert_sample_prediction(postgres_engine, model_id)
+
+    print("[TEST] ✓ Data and previous prediction inserted")
+
+    input_data = get_base_input_data(model_id)
+    input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['policy'] = 'REPEAT'
+
+    print("\n[TEST] 2. Starting workflow that should trigger REPEAT...")
+    workflow_id = f'test-input-repeat-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_repeat(postgres_engine, model_id, data)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+@pytest.fixture
+def bad_data_model(patch_mlflow):
+    model = MagicMock(
+        predict=MagicMock(
+            side_effect=Exception("Bad data model")
+        )
+    )
+
+    patch_mlflow.sklearn.load_model = MagicMock(return_value=model)
+    
+    return model
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_2_1_transform_gate_triggers_continue(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_data_model,
+):
+    """
+    Scenario 2.2.1: Transform Gate Triggers CONTINUE
+    
+    Description:
+    Transform response gate determines data should continue despite issues.
+    
+    Expected Behavior:
+    - request_transform succeeds
+    - mlflow_response_gate for transform returns path_flag='CONTINUE'
+    - path_flag_handler calls export workflow with transform data
+    - MLFlow predict skipped
+    - Transform data exported as-is
+    
+    Assertions:
+    - Transform completed
+    - mlflow_response_gate called for transform
+    - MLFlow predict NOT called
+    - Export workflow called with transform data
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 221
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_transform_filters']['API_ERROR']['policy'] = 'CONTINUE'
+
+    print("\n[TEST] 2. Starting workflow that should trigger CONTINUE at transform gate...")
+    workflow_id = f'test-transform-continue-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_continue(
+        postgres_engine=postgres_engine,
+        model_id=model_id,
+        prediction_confidence=Decimal(10),
+        comments='Bad data model',
+    )
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_2_2_transform_gate_triggers_stop(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_data_model,
+):
+    """
+    Scenario 2.2.2: Transform Gate Triggers STOP
+    
+    Description:
+    Transform response validation fails with STOP policy.
+    
+    Expected Behavior:
+    - request_transform succeeds but response invalid
+    - mlflow_response_gate for transform returns path_flag='STOP'
+    - Workflow exits without calling predict or export
+    
+    Assertions:
+    - Transform completed but validation failed
+    - mlflow_response_gate called for transform
+    - MLFlow predict NOT called
+    - Export workflow NOT called
+    - Workflow completes without error
+    """
+    client = temporal_test_env.client
+
+    model_id = 222
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_transform_filters']['API_ERROR']['policy'] = 'STOP'
+
+    print("\n[TEST] 2. Starting workflow that should trigger STOP at transform gate...")
+    workflow_id = f'test-transform-stop-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_stop(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_2_3_transform_gate_triggers_repeat(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_data_model,
+):
+    """
+    Scenario 2.2.3: Transform Gate Triggers REPEAT
+    
+    Description:
+    Transform response gate determines data should repeat last prediction.
+    
+    Expected Behavior:
+    - request_transform succeeds but response has issues
+    - mlflow_response_gate for transform returns path_flag='REPEAT'
+    - path_flag_handler calls repeat_last_prediction activity
+    - MLFlow predict skipped
+    - Last prediction repeated and exported
+    
+    Assertions:
+    - Transform completed but validation triggered REPEAT
+    - mlflow_response_gate called for transform
+    - MLFlow predict NOT called
+    - repeat_last_prediction activity called
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 223
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+    data = insert_sample_prediction(postgres_engine, model_id)
+
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_transform_filters']['API_ERROR']['policy'] = 'REPEAT'
+
+    print("\n[TEST] 2. Starting workflow that should trigger REPEAT at transform gate...")
+    workflow_id = f'test-transform-repeat-{datetime.now().timestamp()}'
+
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_repeat(postgres_engine, model_id, data)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.fixture
+def bad_predict_model(
+    patch_mlflow,
+    mock_mlflow_models
+):
+    model = MagicMock(
+        predict=MagicMock(
+            side_effect=Exception("Bad predict model")
+        )
+    )
+
+    def mock_sklearn_load_model(model_uri):
+        if 'data_model' in model_uri or 'transform' in model_uri.lower():
+            return mock_mlflow_models['transform_model']
+        return model
+    patch_mlflow.sklearn = MagicMock()
+    patch_mlflow.sklearn.load_model = MagicMock(side_effect=mock_sklearn_load_model)
+
+    return model
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_3_1_predict_gate_triggers_continue(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_predict_model,
+):
+    """
+    Scenario 2.3.1: Predict Gate Triggers CONTINUE
+    
+    Description:
+    Predict response gate determines data should continue despite issues.
+    
+    Expected Behavior:
+    - request_predict succeeds
+    - mlflow_response_gate for predict returns path_flag='CONTINUE'
+    - path_flag_handler calls export workflow with predict data
+    - Prediction exported despite quality issues
+    
+    Assertions:
+    - Transform and predict completed
+    - mlflow_response_gate called for predict
+    - Export workflow called with predict data
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 231
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
+
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_predict_filters']['API_ERROR']['policy'] = 'CONTINUE'
+
+    print("\n[TEST] 2. Starting workflow that should trigger CONTINUE at predict gate...")
+    workflow_id = f'test-predict-continue-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_continue(
+        postgres_engine=postgres_engine,
+        model_id=model_id,
+        prediction_confidence=Decimal(10),
+        comments='Bad predict model',
+    )
+    
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_3_2_predict_gate_triggers_stop(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_predict_model,
+):
+    """
+    Scenario 2.3.2: Predict Gate Triggers STOP
+    
+    Description:
+    Prediction validation fails with STOP policy.
+    
+    Expected Behavior:
+    - request_predict succeeds but response invalid
+    - mlflow_response_gate for predict returns path_flag='STOP'
+    - Workflow exits without export
+    
+    Assertions:
+    - Transform completed
+    - Predict completed but validation failed
+    - Export workflow NOT called
+    - Workflow completes without error
+    """
+    client = temporal_test_env.client
+
+    model_id = 232
+
+    print("\n[TEST] 1. Inserting test data...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    print("[TEST] ✓ Data inserted successfully")
+
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_predict_filters']['API_ERROR']['policy'] = 'STOP'
+
+    print("\n[TEST] 2. Starting workflow that should stop at predict gate...")
+    workflow_id = f'test-predict-stop-{datetime.now().timestamp()}'
+    
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_stop(postgres_engine, model_id)
+
+    print("\n[TEST] ✓ All assertions passed!")
+
+
+@pytest.mark.asyncio
+@pytest.mark.integration
+async def test_scenario_2_3_3_predict_gate_triggers_repeat(
+    temporal_test_env: WorkflowEnvironment,
+    temporal_worker: Worker,
+    test_activities: Activities,
+    postgres_engine,
+    bad_predict_model,
+):
+    """
+    Scenario 2.3.3: Predict Gate Triggers REPEAT
+    
+    Description:
+    Predict response gate determines data should repeat last prediction.
+    
+    Expected Behavior:
+    - request_predict succeeds but response has issues
+    - mlflow_response_gate for predict returns path_flag='REPEAT'
+    - path_flag_handler calls repeat_last_prediction activity
+    - Last prediction repeated and exported
+    
+    Assertions:
+    - Transform and predict completed but validation triggered REPEAT
+    - mlflow_response_gate called for predict
+    - repeat_last_prediction activity called
+    - Export workflow NOT called with current prediction
+    - Workflow completes
+    """
+    client = temporal_test_env.client
+
+    model_id = 233
+
+    print("\n[TEST] 1. Inserting test data and previous prediction...")
+    insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
+    data = insert_sample_prediction(postgres_engine, model_id)
+
+    print("[TEST] ✓ Data and previous prediction inserted")
+
+    input_data = get_base_input_data(model_id)
+    input_data['mlflow_predict_filters']['API_ERROR']['policy'] = 'REPEAT'
+    input_data['path_priority'] = ['REPEAT', 'STOP', 'CONTINUE']  # REPEAT first
+
+    print("\n[TEST] 2. Starting workflow that should trigger REPEAT at predict gate...")
+    workflow_id = f'test-predict-repeat-{datetime.now().timestamp()}'
+
+    await start_and_await_workflow(client, input_data, workflow_id)
+    assert_repeat(postgres_engine, model_id, data)
+
+    print("\n[TEST] ✓ All assertions passed!")

git-requirements-mapping.txt

@@ -0,0 +1,2 @@
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git:sientia-do
+git+ssh://git@github.com/Aignosi/sientia-mlops-library.git:sientia

laborious/activities/activities.py

@@ -7,6 +7,7 @@ with workflow.unsafe.imports_passed_through():
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.observability.logger import Logger
 
+    from laborious.activities.api import API
     from laborious.activities.gates import Gates
     from laborious.activities.mlflow import MLFlow
     from laborious.activities.model_metrics import ModelMetrics
@@ -14,7 +15,7 @@ with workflow.unsafe.imports_passed_through():
     from laborious.activities.storage import Storage
 
 
-class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
+class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
     """
     Main activities orchestrator for the Laborious system.
 
@@ -23,15 +24,18 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
     MLFlow model interactions, data quality validation, and OPC server communications.
 
     The class implements multiple inheritance to combine specialized functionality:
-    - Postgres: Database operations and data persistence
+    - Storage: Database operations and data persistence
     - MLFlow: Model inference and transformation operations
     - Gates: Data quality validation and filtering mechanisms
     - OPC: Real-time data export to OPC servers
+    - ModelMetrics: Model performance metrics and drift detection
+    - API: PI Web API export operations for industrial systems
 
     Attributes:
         postgres_config (dict): PostgreSQL connection configuration
         mlflow_config (dict): MLFlow server configuration
         opc_config (dict): OPC server configuration
+        pi_web_api_config (dict): PI Web API server configuration
         logger (Logger): Logging and observability instance
         notification_handler (NotificationHandler): Notification management instance
     """
@@ -42,6 +46,7 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
         mlflow_config: dict[str, Any],
         minio_config: dict[str, Any],
         opc_config: dict[str, Any],
+        pi_web_api_config: dict[str, Any],
         logger: Logger,
         notification_handler: NotificationHandler,
     ):
@@ -58,6 +63,8 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
                 Required keys: host, port, username, password
             opc_config: OPC server configuration dictionary
                 Can contain multiple server configurations
+            pi_web_api_config: PI Web API server configuration dictionary
+                Required keys: base_url, auth_type, auth_token
             logger: Logger instance for observability and debugging
             notification_handler: Notification handler for alerts and monitoring
 
@@ -116,6 +123,16 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
             metrics_controller=metrics_controller,
         )
 
+        API.__init__(
+            self,
+            base_url=pi_web_api_config['base_url'],
+            auth_type=pi_web_api_config['auth_type'],
+            auth_token=pi_web_api_config['auth_token'],
+            logger=logger,
+            notification_handler=notification_handler,
+            metrics_controller=metrics_controller,
+        )
+
     async def shutdown(self):
         """
         Gracefully shutdown all activities and clean up resources.
@@ -123,6 +140,8 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
         This method ensures proper cleanup of all resources including:
         - PostgreSQL connection pools
         - OPC server connections
+        - PI Web API client connections
+        - MLFlow model repositories
         - Any other resources that need explicit cleanup
 
         The method should be called before the application terminates to ensure
@@ -133,3 +152,4 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
         Gates.close(self)
         await OPC.close(self)
         ModelMetrics.close(self)
+        API.close(self)

laborious/activities/api.py

@@ -0,0 +1,270 @@
+from temporalio import activity, workflow
+
+with workflow.unsafe.imports_passed_through():
+    import json
+    import traceback
+    from typing import Any
+
+    from pandas import DataFrame
+    from sientia_do.notifications.handlers import NotificationHandler
+    from sientia_do.notifications.models import NotificationLevel
+    from sientia_do.observability.logger import Logger
+    from sientia_do.observability.metrics_controller import MetricsController
+    from sientia_do.observability.sientia_monitoring import SientiaMonitoring
+    from sientia_do.repository.pi_web_api_client import PIWebAPIClient
+
+    from laborious import metrics
+
+
+PI_WEB_API_PREDICTION_ERROR_CONFIDENCE = 13
+
+
+class API(SientiaMonitoring):
+    """
+    PI Web API operations for writing prediction data to PI Web API.
+
+    This class provides Temporal activities for interacting with the PI Web API
+    to write prediction and confidence values to industrial systems. It handles
+    error scenarios gracefully by setting error confidence values and sending
+    notifications when write operations fail.
+
+    The class implements comprehensive error handling for both prediction and
+    confidence value writes, ensuring that partial failures are properly
+    reported and handled.
+    """
+
+    def __init__(
+        self,
+        base_url: str,
+        auth_type: str,
+        auth_token: str,
+        logger: Logger,
+        notification_handler: NotificationHandler,
+        metrics_controller: MetricsController,
+    ) -> None:
+        """
+        Initialize API activity with PI Web API client.
+
+        Args:
+            base_url (str): Base URL of the PI Web API server
+            auth_type (str): Authentication type ('basic' or 'bearer')
+            auth_token (str): Authentication token
+            logger (Logger): Logger instance for operation logging
+            notification_handler (NotificationHandler): Handler for system notifications
+            metrics_controller (MetricsController): Controller for metrics collection
+        """
+        SientiaMonitoring.__init__(self, logger, notification_handler, metrics_controller)
+        self.pi_web_api_client = PIWebAPIClient(
+            base_url=base_url,
+            auth_config={
+                'type': auth_type,
+                'token': auth_token,
+            },
+            logger=logger,
+            notification_handler=notification_handler,
+            metrics_controller=metrics_controller,
+        )
+
+    def close(self) -> None:
+        """
+        Close the PI Web API client and shutdown monitoring services.
+
+        This method properly closes all connections and resources associated
+        with the PI Web API client and monitoring services.
+        """
+        self.pi_web_api_client.close()
+        SientiaMonitoring.shutdown(self)
+
+    async def process_pi_web_api_response(
+        self,
+        response_data: list[dict[str, Any]],
+        tags: dict[str, str],
+        core_labels: dict[str, str],
+        metadata: dict[str, Any],
+    ) -> tuple[int, str]:
+        """
+        Process the response data from PI Web API write operation.
+
+        Validates that all tags were successfully written, emits metrics for each tag
+        (success or error), and returns the appropriate prediction confidence value.
+        Sets error confidence if any tag write fails or if the number of written tags
+        doesn't match the expected count.
+
+        Args:
+            - response_data (dict[str, Any]): The response data from the PI Web API write operation.
+            - tags (dict[str, str]): The tags that were written to the PI Web API.
+            - core_labels (dict[str, str]): The core labels of the workflow execution.
+            - metadata (dict[str, Any]): The metadata of the workflow execution.
+        Returns:
+            int: Prediction confidence value (0 for success, 13 for errors)
+        """
+
+        # Convert tags from name:webid to webid:name
+        tags = {w: t for t, w in tags.items()}
+
+        tag_names = list[str](tags.values())
+
+        confidence = 0
+
+        message = ''
+
+        # Evaluate response for each tag
+        written_tags = []
+        for item in response_data:
+            web_id = item.get('WebId')
+            if not web_id:
+                self.error('The response did not contain some WebIds', metadata)
+                continue
+            errors = item.get('Errors', [])
+            tag_name = tags.get(web_id)
+            if not tag_name:
+                self.error(
+                    f'The response did not contain the tag name for WebId {web_id}', metadata
+                )
+                continue
+            if errors:
+                self.error(
+                    f'Error writing tag {tag_name}:{web_id} to PI Web API: {errors}', metadata
+                )
+                await self.emit_metric(
+                    metric_object=metrics.PI_WEB_API_PREDICTION_WRITTEN_ERROR_COUNT,
+                    tags={
+                        **core_labels,
+                        'tag_name': tag_name,
+                    },
+                )
+                confidence = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+            else:
+                await self.emit_metric(
+                    metric_object=metrics.PI_WEB_API_PREDICTION_WRITTEN_COUNT,
+                    tags={
+                        **core_labels,
+                        'tag_name': tag_name,
+                    },
+                )
+                written_tags.append(tag_name)
+
+        if len(written_tags) != len(tag_names):
+            message = f'The number of written tags does not match the number of tag names: Expected {tag_names} tags, but {written_tags} tags were written.'
+
+            self.error(
+                f'{message}\nResponse:\n {json.dumps(response_data, indent=4)}\nTags:\n {json.dumps(tags, indent=4)}',
+                metadata,
+            )
+
+            await self.send_notification_async(
+                metadata=metadata,
+                notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
+                message=f'The number of written tags does not match the number of tag names: Expected {tag_names} tags, but {written_tags} tags were written.\nResponse:\n {json.dumps(response_data, indent=4)}\nTags:\n {json.dumps(tags, indent=4)}',
+                block='write_pi_web_api_data',
+                level=NotificationLevel.ERROR,
+            )
+            confidence = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+
+        return confidence, message
+
+    @activity.defn(name='write_pi_web_api_data')
+    async def write_pi_web_api_data(self, input_data: dict[str, Any]) -> dict[Any, Any]:
+        """
+        Write prediction and confidence data to PI Web API.
+
+        Writes prediction values and confidence scores to PI Web API using configured
+        web IDs. Processes responses to validate writes and emit metrics. Handles errors
+        gracefully by setting error confidence values when writes fail and sending
+        notifications for both prediction and confidence write errors.
+
+        Args:
+            input_data (dict[str, Any]): The input data containing:
+                - metadata (dict[str, Any]): Workflow execution metadata
+                - pi_web_api_output_config (dict[str, Any]): PI Web API configuration with:
+                    - endpoint (str): PI Web API endpoint URL
+                    - prediction_tags (dict[str, str]): Mapping of tag names to web IDs for predictions
+                    - confidence_tags (dict[str, str]): Mapping of tag names to web IDs for confidence
+                - data (dict[str, Any]): Prediction data, its a dataframe converted to dict.
+        Returns:
+            dict[Any, Any]: Data dictionary with potentially modified confidence values
+                If prediction write fails, prediction_confidence is set to error value (13)
+        """
+        metadata = input_data['metadata']
+        data = DataFrame(input_data['data'])
+        pi_web_api_output_config = input_data['pi_web_api_output_config']
+
+        self.info(f'Writing data to PI Web API... config: {pi_web_api_output_config}', metadata)
+
+        raw_prediction_tags = pi_web_api_output_config['prediction_tags']
+        raw_confidence_tags = pi_web_api_output_config['confidence_tags']
+        prediction_tags = list[str](raw_prediction_tags.values())
+        confidence_tags = list(raw_confidence_tags.values())
+
+        core_labels = self.get_core_labels(metadata)
+
+        prediction_value = data.head(1)['prediction'].values[0]
+        confidence_value = data.head(1)['prediction_confidence'].values[0]
+
+        try:
+            prediction_response = await self.pi_web_api_client.write_value(
+                web_ids=prediction_tags,
+                value={
+                    'Timestamp': data.head(1)['timestamp'].values[0],
+                    'Value': prediction_value,
+                },
+                metadata=metadata,
+            )
+
+            confidence, message = await self.process_pi_web_api_response(
+                response_data=prediction_response,
+                tags=raw_prediction_tags,
+                core_labels=core_labels,
+                metadata=metadata,
+            )
+
+            data['prediction_confidence'] = confidence
+            data['comments'] = message
+
+        except Exception as e:
+            trace = traceback.format_exc()
+            await self.send_notification_async(
+                metadata=metadata,
+                notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
+                message=f'Error writing prediction data to PI Web API: {e}\n Tags: {raw_prediction_tags}',
+                block='write_pi_web_api_data',
+                level=NotificationLevel.ERROR,
+                attachment_content=trace,
+            )
+
+            self.error(trace, metadata)
+
+            data['prediction_confidence'] = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+            data['comments'] = str(e)
+
+            return data.to_dict()
+
+        try:
+            confidence_response = await self.pi_web_api_client.write_value(
+                web_ids=confidence_tags,
+                value={
+                    'Timestamp': data.head(1)['timestamp'].values[0],
+                    'Value': confidence_value,
+                },
+                metadata=metadata,
+            )
+
+            await self.process_pi_web_api_response(
+                response_data=confidence_response,
+                tags=raw_confidence_tags,
+                core_labels=core_labels,
+                metadata=metadata,
+            )
+
+        except Exception as e:
+            trace = traceback.format_exc()
+            await self.send_notification_async(
+                metadata=metadata,
+                notification_id='WRITE_PI_WEB_API_CONFIDENCE_ERROR',
+                message=f'Error writing confidence data to PI Web API: {e}\n Tags: {raw_confidence_tags}',
+                block='write_pi_web_api_data',
+                level=NotificationLevel.ERROR,
+                attachment_content=trace,
+            )
+
+        return data.to_dict()

laborious/activities/gates.py

@@ -6,13 +6,13 @@ with workflow.unsafe.imports_passed_through():
     from typing import Any
 
     from pandas import DataFrame
-    from sientia_do.formatters import create_sample_dict
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.logger import Logger
     from sientia_do.observability.metrics_controller import MetricsController
     from sientia_do.observability.sientia_monitoring import SientiaMonitoring
     from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ, now
+    from sientia_do.utils.formatters import create_sample_dict
 
     from laborious import metrics
     from laborious.utils.filters.conditional_filters import (

laborious/activities/mlflow.py

@@ -6,7 +6,6 @@ with workflow.unsafe.imports_passed_through():
 
     import numpy as np
     from pandas import DataFrame, to_datetime
-    from sientia_do.formatters import create_sample_dict
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.notifications.models import NotificationLevel
     from sientia_do.observability.logger import Logger
@@ -18,6 +17,7 @@ with workflow.unsafe.imports_passed_through():
         DATETIME_FORMAT_WITH_TZ,
         now,
     )
+    from sientia_do.utils.formatters import create_sample_dict
 
     from laborious.utils.repository.minio_repository import MinioRepository
     from laborious.utils.repository.model_repository import MLFlowRepository
@@ -153,11 +153,11 @@ class MLFlow(SientiaMonitoring):
         # Pivot data for model input format
         data = data.pivot(index='timestamp', columns='variable', values='value')
         data.fillna(np.nan, inplace=True)
-        # data.reset_index(inplace=True)
-        data.columns.name = None
 
-        self.debug('Processed input data:', metadata)
+        data.columns.name = None
-        self.debug(data.head(5).to_string(), metadata)
+        data.index.name = None
+
+        self.debug(f'Processed input data: \n {data.to_csv()}', metadata)
 
         # Request transformation from MLFlow model
         response_data = await self.model_monitoring_repository.transform(

laborious/activities/opc.py

@@ -352,10 +352,13 @@ class OPC(SientiaMonitoring):
             This allows downstream systems to handle data quality appropriately.
         """
 
+        message = 'Some data could not be written to OPC servers'
+
         if not success:
             data['prediction_confidence'] = OPC_WRITTING_ERROR_CONFIDENCE
+            data['comments'] = message
             self.debug(
-                f'Some data could not be written to OPC servers, setting confidence to  {OPC_WRITTING_ERROR_CONFIDENCE}.',
+                f'{message}, setting confidence to  {OPC_WRITTING_ERROR_CONFIDENCE}.',
                 metadata,
             )
 

laborious/metrics.py

@@ -24,7 +24,9 @@ Metric Labels:
 """
 
 from prometheus_client import Counter, Gauge, Histogram
-from sientia_do.observability.metrics import CORE_LABELS as SIENTIA_CORE_LABELS
+from sientia_do.observability.metrics import (
+    CORE_LABELS as SIENTIA_CORE_LABELS,
+)
 
 # Application health metric
 APP_UP = Gauge(
@@ -188,3 +190,20 @@ MODEL_ANALYZE_ERROR_COUNT = Counter(
     'Number of errors during analyze operations',
     SIENTIA_CORE_LABELS,
 )
+
+
+# ================== PI Web API metrics ==================
+
+PI_WEB_API_LABELS = [*CORE_LABELS, 'tag_name']
+
+PI_WEB_API_PREDICTION_WRITTEN_COUNT = Counter(
+    'laborious_pi_web_api_prediction_written_count',
+    'Number of predictions written to the PI Web API',
+    PI_WEB_API_LABELS,
+)
+
+PI_WEB_API_PREDICTION_WRITTEN_ERROR_COUNT = Counter(
+    'laborious_pi_web_api_prediction_written_error_count',
+    'Number of errors writing predictions to the PI Web API',
+    PI_WEB_API_LABELS,
+)

laborious/utils/connectors_config.py

@@ -3,37 +3,6 @@ from os import getenv
 from typing import Any
 
 
-def build_postgres_config() -> dict[str, Any]:
-    """
-    Build PostgreSQL database configuration from environment variables.
-
-    This function constructs a PostgreSQL configuration dictionary from
-    environment variables with sensible defaults for local development.
-    It handles connection pool configuration and security parameters.
-
-    Environment Variables:
-        POSTGRES_HOST: Database hostname (default: localhost)
-        POSTGRES_PORT: Database port (default: 5432)
-        POSTGRES_USER: Database username (default: sientia)
-        POSTGRES_PASSWORD: Database password (default: sientia)
-        POSTGRES_DBNAME: Database name (default: sientia)
-        POSTGRES_MIN_CONNECTIONS: Minimum connection pool size (default: 5)
-        POSTGRES_MAX_CONNECTIONS: Maximum connection pool size (default: 20)
-
-    Returns:
-        dict: PostgreSQL configuration dictionary with all required parameters
-    """
-    return {
-        'host': getenv('POSTGRES_HOST', 'localhost'),
-        'port': int(getenv('POSTGRES_PORT', '5432')),
-        'user': getenv('POSTGRES_USER', 'sientia'),
-        'password': getenv('POSTGRES_PASSWORD', 'sientia'),
-        'dbname': getenv('POSTGRES_DBNAME', 'sientia'),
-        'min_connections': int(getenv('POSTGRES_MIN_CONNECTIONS', '5')),
-        'max_connections': int(getenv('POSTGRES_MAX_CONNECTIONS', '20')),
-    }
-
-
 def build_mlflow_config() -> dict[str, Any]:
     """
     Build MLFlow server configuration from environment variables.
@@ -99,37 +68,6 @@ def build_opc_config() -> dict[str, Any]:
     }
 
 
-def build_mongodb_config() -> dict[str, Any]:
-    """
-    Build MongoDB configuration from environment variables.
-
-    This function constructs a MongoDB configuration dictionary from
-    environment variables with sensible defaults for local development.
-    It handles connection string and database name configuration.
-
-    Environment Variables:
-        MONGODB_USERNAME: MongoDB username (default: root)
-        MONGODB_PASSWORD: MongoDB password (default: wKZDbMNU1c)
-        MONGODB_URL: MongoDB connection URI (default: localhost:27018)
-        MONGODB_DATABASE_NAME: MongoDB database name (default: sientia)
-        MONGODB_TTL_INDEX_HOURS: TTL index duration in hours (default: 1)
-
-    Returns:
-        dict: MongoDB configuration dictionary with connection parameters
-    """
-    username = getenv('MONGODB_USERNAME', 'root')
-    password = getenv('MONGODB_PASSWORD', 'wKZDbMNU1c')
-    uri = getenv('MONGODB_URL', 'localhost:27018')
-
-    connection_string = f'mongodb://{username}:{password}@{uri}'
-
-    return {
-        'connection_string': connection_string,
-        'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
-        'ttl_index_seconds': int(getenv('MONGODB_TTL_INDEX_HOURS', '1')) * 3600,
-    }
-
-
 def build_minio_config() -> dict[str, Any]:
     """
     Build MinIO (S3-compatible) configuration from environment variables.

laborious/utils/repository/model_repository.py

@@ -752,6 +752,7 @@ class MLFlowRepository(SientiaMonitoring):
         latest_production_id: str,
         metadata: dict,
         transform_flavor: str = 'sklearn',
+        skip_transform: bool = False,
         predict_flavor: str = 'sklearn',
         target_name: str | None = None,
     ) -> dict[str, Any]:
@@ -812,7 +813,10 @@ class MLFlowRepository(SientiaMonitoring):
             load_wrapper=load_predict_wrapper,
         )
 
-        treated_data_candidate = data_model.fit(data)
+        if not skip_transform:
+            treated_data_candidate = data_model.fit(data)
+        else:
+            treated_data_candidate = data_model
 
         if not isinstance(treated_data_candidate, pd.DataFrame):
             data_model = treated_data_candidate
@@ -1164,7 +1168,7 @@ class MLFlowRepository(SientiaMonitoring):
                       and returned in the response structure rather than propagated.
         """
 
-        self.debug(f'Data received for model transformation: {data.head(5).to_csv()}', metadata)
+        self.debug(f'Data received for model transformation: {data.to_csv()}', metadata)
 
         # data.to_csv(
         #     f"tmp/data_{model_name}.csv", index=True)
@@ -1250,7 +1254,9 @@ class MLFlowRepository(SientiaMonitoring):
             input_index = data.index
             start_time = datetime.now()
 
-            self.debug(f'Data received for model prediction: {data.head(5).to_csv()}', metadata)
+            self.debug(
+                f'Data received for model prediction: {data.to_dict(orient="records")}', metadata
+            )
 
             # data.to_csv(
             #     f"tmp/treated_data_{model_name}.csv", index=True)
@@ -1268,7 +1274,8 @@ class MLFlowRepository(SientiaMonitoring):
 
             if isinstance(predict_data, pd.DataFrame):
                 self.debug(
-                    f'Data received from model prediction: {data.head(5).to_csv()}', metadata
+                    f'Data received from model prediction: {predict_data.to_dict(orient="records")}',
+                    metadata,
                 )
 
                 # predict_data.to_csv(
@@ -1343,6 +1350,7 @@ class MLFlowRepository(SientiaMonitoring):
 
         transform_flavor = model_config.get('transform_flavor', 'sklearn')
         predict_flavor = model_config.get('predict_flavor', 'sklearn')
+        skip_transform = model_config.get('skip_transform', False)
 
         self.debug(
             f'Model configuration - transform_flavor: {transform_flavor}, predict_flavor: {predict_flavor}, target_name: {target_name}',
@@ -1356,6 +1364,7 @@ class MLFlowRepository(SientiaMonitoring):
                 model_name=model_name,
                 data=data,
                 transform_flavor=transform_flavor,
+                skip_transform=skip_transform,
                 predict_flavor=predict_flavor,
                 target_name=target_name,
                 metadata=metadata,

laborious/worker/prepare_worker.py

@@ -0,0 +1,72 @@
+import os
+import re
+from collections.abc import Sequence
+from typing import Any
+
+from sientia_do.observability.logger import Logger
+from temporalio.client import Client
+from temporalio.worker import PollerBehaviorAutoscaling, Worker
+
+parameters = [
+    ('MAX_CONCURRENT_WORKFLOW_TASKS', '200'),
+    ('MAX_CONCURRENT_ACTIVITIES', '200'),
+    ('MAX_CONCURRENT_LOCAL_ACTIVITIES', '200'),
+    ('MAX_CACHED_WORKFLOWS', '200'),
+    ('WORKFLOW_POLLER_BEHAVIOUR_MINIMUM', '10'),
+    ('WORKFLOW_POLLER_BEHAVIOUR_INITIAL', '100'),
+    ('WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM', '200'),
+    ('ACTIVITY_POLLER_BEHAVIOUR_MINIMUM', '10'),
+    ('ACTIVITY_POLLER_BEHAVIOUR_INITIAL', '100'),
+    ('ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM', '200'),
+]
+
+
+def camel_to_snake(text: str) -> str:
+    """Convert camelCase or PascalCase to snake_case."""
+    text = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', text)
+    text = re.sub('([a-z0-9])([A-Z])', r'\1_\2', text)
+    return text.lower()
+
+
+def prepare_worker(
+    main_workflow: type,
+    other_workflows: Sequence[type],
+    activities: Sequence[Any],
+    temporal_client: Client,
+    logger: Logger,
+) -> Worker:
+    main_workflow_name = main_workflow.__name__.upper()
+
+    queue_name = f'{camel_to_snake(main_workflow.__name__)}-queue'
+
+    local_workflow_parameters = {}
+
+    for parameter in parameters:
+        local_workflow_parameters[parameter[0]] = int(
+            os.getenv(main_workflow_name + '_' + parameter[0], parameter[1])
+        )
+
+    logger.info(f'Preparing worker for {main_workflow_name} with queue {queue_name}')
+
+    return Worker(
+        temporal_client,
+        task_queue=queue_name,
+        workflows=[main_workflow, *other_workflows],
+        activities=[*activities],
+        max_concurrent_workflow_tasks=local_workflow_parameters['MAX_CONCURRENT_WORKFLOW_TASKS'],
+        max_concurrent_activities=local_workflow_parameters['MAX_CONCURRENT_ACTIVITIES'],
+        max_concurrent_local_activities=local_workflow_parameters[
+            'MAX_CONCURRENT_LOCAL_ACTIVITIES'
+        ],
+        max_cached_workflows=local_workflow_parameters['MAX_CACHED_WORKFLOWS'],
+        workflow_task_poller_behavior=PollerBehaviorAutoscaling(
+            minimum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MINIMUM'],
+            initial=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_INITIAL'],
+            maximum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM'],
+        ),
+        activity_task_poller_behavior=PollerBehaviorAutoscaling(
+            minimum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MINIMUM'],
+            initial=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_INITIAL'],
+            maximum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM'],
+        ),
+    )

laborious/worker/worker.py

@@ -7,6 +7,7 @@ prediction and retraining workflows.
 
 The worker supports multiple task queues:
 - predictions_batch-queue: Handles batch prediction workflows (heavy workload)
+  Includes activities for MLFlow, data quality gates, OPC export, PI Web API export, and PostgreSQL
 - minimal_retrain-queue: Handles model retraining workflows
 - drift-queue: Handles drift detection workflows
 - simple_metrics-queue: Handles simple metrics calculation workflows
@@ -26,51 +27,33 @@ Environment Variables:
 - HTTP_METRICS_PORT: Prometheus metrics server port (default: 9090)
 - HTTP_SDK_METRICS_PORT: Temporal SDK metrics port (default: 9091)
 - PROJECT_NAME: Project name for notifications (default: laborious)
-
-Tuner Configuration (Resource-based scaling):
-- TUNER_TARGET_MEMORY_USAGE: Target memory usage (0.0-1.0, default: 0.75)
-- TUNER_TARGET_CPU_USAGE: Target CPU usage (0.0-1.0, default: 0.80)
-- TUNER_WORKFLOW_MIN_SLOTS: Minimum workflow slots (default: 5)
-- TUNER_WORKFLOW_MAX_SLOTS: Maximum workflow slots (default: 50)
-- TUNER_ACTIVITY_MIN_SLOTS: Minimum activity slots (default: 5)
-- TUNER_ACTIVITY_MAX_SLOTS: Maximum activity slots (default: 50)
-- TUNER_WORKFLOW_RAMP_THROTTLE_MS: Workflow ramp throttle in ms (default: 100)
-- TUNER_ACTIVITY_RAMP_THROTTLE_MS: Activity ramp throttle in ms (default: 50)
-
-Poller Configuration:
-- POLLER_MINIMUM: Minimum number of pollers (default: 1)
-- POLLER_MAXIMUM: Maximum number of pollers (default: 10)
-- POLLER_INITIAL: Initial number of pollers (default: 2)
 """
 
-
 from temporalio import client, workflow
 from temporalio.runtime import PrometheusConfig, Runtime, TelemetryConfig
-from temporalio.worker import (
-    PollerBehaviorAutoscaling,
-    ResourceBasedSlotConfig,
-    Worker,
-    WorkerTuner,
-)
 
 with workflow.unsafe.imports_passed_through():
     import asyncio
+    import os
     import sys
-    from datetime import timedelta
 
     from prometheus_client import start_http_server
     from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
     from sientia_do.observability.logger import get_logger
+    from sientia_do.utils.connectors_config import (
+        build_api_config,
+        build_mongodb_config,
+        build_postgres_config,
+    )
 
     from laborious import metrics
     from laborious.activities.activities import Activities
     from laborious.utils.connectors_config import (
         build_minio_config,
         build_mlflow_config,
-        build_mongodb_config,
         build_opc_config,
-        build_postgres_config,
     )
+    from laborious.worker.prepare_worker import prepare_worker
     from laborious.workflows.drift import Drift
     from laborious.workflows.minimal_retrain import MinimalRetrain
     from laborious.workflows.predictions_batch import PredictionsBatch
@@ -79,55 +62,11 @@ with workflow.unsafe.imports_passed_through():
         FormatAndExportPrediction,
     )
     from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
-    import os
 
 POD_ID = os.getenv('POD_ID')
 SDK_METRICS_PORT = int(os.getenv('HTTP_SDK_METRICS_PORT', '9091'))
 
 
-def create_resource_tuner() -> WorkerTuner:
-    """Create a resource-based tuner from environment variables."""
-    target_memory = float(os.getenv('TUNER_TARGET_MEMORY_USAGE', '0.75'))
-    target_cpu = float(os.getenv('TUNER_TARGET_CPU_USAGE', '0.50'))
-    workflow_min = int(os.getenv('TUNER_WORKFLOW_MIN_SLOTS', '5'))
-    workflow_max = int(os.getenv('TUNER_WORKFLOW_MAX_SLOTS', '50'))
-    activity_min = int(os.getenv('TUNER_ACTIVITY_MIN_SLOTS', '5'))
-    activity_max = int(os.getenv('TUNER_ACTIVITY_MAX_SLOTS', '50'))
-    local_activity_min = int(os.getenv('TUNER_LOCAL_ACTIVITY_MIN_SLOTS', '1'))
-    local_activity_max = int(os.getenv('TUNER_LOCAL_ACTIVITY_MAX_SLOTS', '30'))
-    workflow_ramp = int(os.getenv('TUNER_WORKFLOW_RAMP_THROTTLE_MS', '100'))
-    activity_ramp = int(os.getenv('TUNER_ACTIVITY_RAMP_THROTTLE_MS', '50'))
-    local_activity_ramp = int(os.getenv('TUNER_LOCAL_ACTIVITY_RAMP_THROTTLE_MS', '50'))
-
-    return WorkerTuner.create_resource_based(
-        target_memory_usage=target_memory,
-        target_cpu_usage=target_cpu,
-        workflow_config=ResourceBasedSlotConfig(
-            minimum_slots=workflow_min,
-            maximum_slots=workflow_max,
-            ramp_throttle=timedelta(milliseconds=workflow_ramp),
-        ),
-        activity_config=ResourceBasedSlotConfig(
-            minimum_slots=activity_min,
-            maximum_slots=activity_max,
-            ramp_throttle=timedelta(milliseconds=activity_ramp),
-        ),
-        local_activity_config=ResourceBasedSlotConfig(
-            minimum_slots=local_activity_min,
-            maximum_slots=local_activity_max,
-            ramp_throttle=timedelta(milliseconds=local_activity_ramp),
-        ),
-    )
-
-
-def create_poller_behavior() -> PollerBehaviorAutoscaling:
-    """Create poller behavior from environment variables."""
-    minimum = int(os.getenv('POLLER_MINIMUM', '1'))
-    maximum = int(os.getenv('POLLER_MAXIMUM', '10'))
-    initial = int(os.getenv('POLLER_INITIAL', '2'))
-    return PollerBehaviorAutoscaling(minimum=minimum, maximum=maximum, initial=initial)
-
-
 async def main():
     """
     Main entry point for the Laborious worker application.
@@ -181,6 +120,7 @@ async def main():
         mlflow_config=build_mlflow_config(),
         minio_config=build_minio_config(),
         opc_config=build_opc_config(),
+        pi_web_api_config=build_api_config(),
         logger=logger,
         notification_handler=notification_handler,
     )
@@ -206,14 +146,11 @@ async def main():
 
     logger.custom_info('Starting Workers...', metadata)
 
-    tuner = create_resource_tuner()
-    poller = create_poller_behavior()
-
     workers = [
-        Worker(
+        prepare_worker(
-            temporal_client,
+            temporal_client=temporal_client,
-            task_queue='minimal_retrain-queue',
+            main_workflow=MinimalRetrain,
-            workflows=[MinimalRetrain],
+            other_workflows=[],
             activities=[
                 activities.load_custom_query,
                 activities.query_to_minio,
@@ -222,44 +159,35 @@ async def main():
                 activities.format_retrain_report,
                 activities.export_data_to_postgres,
             ],
-            tuner=tuner,
+            logger=logger,
-            max_cached_workflows=2,
-            workflow_task_poller_behavior=poller,
-            activity_task_poller_behavior=poller,
         ),
-        Worker(
+        prepare_worker(
-            temporal_client,
+            temporal_client=temporal_client,
-            task_queue='drift-queue',
+            main_workflow=SimpleMetrics,
-            workflows=[Drift],
+            other_workflows=[],
+            activities=[
+                activities.load_custom_query,
+                activities.calculate_simple_metrics,
+                activities.export_data_to_postgres,
+            ],
+            logger=logger,
+        ),
+        prepare_worker(
+            temporal_client=temporal_client,
+            main_workflow=Drift,
+            other_workflows=[],
             activities=[
                 activities.load_custom_query,
                 activities.get_reference_data,
                 activities.calculate_drift,
                 activities.export_data_to_postgres,
             ],
-            tuner=tuner,
+            logger=logger,
-            max_cached_workflows=2,
-            workflow_task_poller_behavior=poller,
-            activity_task_poller_behavior=poller,
         ),
-        Worker(
+        prepare_worker(
-            temporal_client,
+            temporal_client=temporal_client,
-            task_queue='simple_metrics-queue',
+            main_workflow=PredictionsBatch,
-            workflows=[SimpleMetrics],
+            other_workflows=[PredictionProcess, FormatAndExportPrediction],
-            activities=[
-                activities.load_custom_query,
-                activities.calculate_simple_metrics,
-                activities.export_data_to_postgres,
-            ],
-            tuner=tuner,
-            max_cached_workflows=2,
-            workflow_task_poller_behavior=poller,
-            activity_task_poller_behavior=poller,
-        ),
-        Worker(
-            temporal_client,
-            task_queue='predictions_batch-queue',
-            workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
             activities=[
                 # MLFlow
                 activities.request_predict,
@@ -279,11 +207,10 @@ async def main():
                 activities.repeat_last_prediction,
                 activities.export_data_to_postgres,
                 activities.write_metrics,
+                # Pi Web API
+                activities.write_pi_web_api_data,
             ],
-            tuner=tuner,
+            logger=logger,
-            max_cached_workflows=200,
-            workflow_task_poller_behavior=poller,
-            activity_task_poller_behavior=poller,
         ),
     ]
 

laborious/workflows/predictions_batch.py

@@ -61,7 +61,10 @@ class PredictionsBatch:
                     - model_retention (int, optional): Model retention period in minutes
                     - path_priority (list[str]): Decision path priority configuration
                     - opc_output_config (dict, optional): OPC server export configuration
+                    - pi_web_api_output_config (dict, optional): PI Web API export configuration
                     - datetime_columns (list[str], optional): Columns to treat as datetime
+                    - save_transform (bool, optional): Whether to save transformed data (default: True)
+                    - prediction_store_policy (str, optional): Data retention policy (default: 'lts:1')
 
         Returns:
             None: The workflow completes successfully when the child workflow finishes
@@ -111,7 +114,9 @@ class PredictionsBatch:
             'model_config': input_data.get('model_config', {}),
             'path_priority': input_data.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT']),
             'opc_output_config': input_data.get('opc_output_config', {}),
+            'pi_web_api_output_config': input_data.get('pi_web_api_output_config', {}),
             'prediction_store_policy': input_data.get('prediction_store_policy', 'lts:1'),
+            'save_transform': input_data.get('save_transform', True),
         }
 
         # Execute prediction process workflow

laborious/workflows/sub_workflows/format_and_export_prediction.py

@@ -26,6 +26,7 @@ class FormatAndExportPrediction:
 
     Export Destinations:
     - PostgreSQL Database: Persistent storage with timestamp conversion
+    - PI Web API: Real-time industrial system integration for prediction and confidence values
     - OPC Servers: Real-time industrial system integration
     - Prometheus Metrics: Performance monitoring and operational visibility
     """
@@ -38,9 +39,10 @@ class FormatAndExportPrediction:
         This method orchestrates the complete data export process by:
         1. Determining the appropriate formatting strategy based on path_flag
         2. Formatting prediction data according to quality and requirements
-        3. Exporting data to OPC servers for real-time industrial access
+        3. Exporting data to PI Web API for real-time industrial access (if configured)
-        4. Persisting data to PostgreSQL database with comprehensive metadata
+        4. Exporting data to OPC servers for real-time industrial access (if configured)
-        5. Recording performance metrics for operational monitoring
+        5. Persisting data to PostgreSQL database with comprehensive metadata
+        6. Recording performance metrics for operational monitoring
 
         The method implements flexible formatting strategies:
         - Normal predictions: Full data formatting with confidence scores
@@ -61,8 +63,10 @@ class FormatAndExportPrediction:
                     - model_name (str): Name of the ML model
                     - schema (str): Database schema for data storage
                     - table_name (str): Target table for data persistence
-                    - opc_output_config (dict[str, Any]): OPC server export configuration
                 Optional keys:
+                    - opc_output_config (dict[str, Any]): OPC server export configuration
+                    - pi_web_api_output_config (dict[str, Any]): PI Web API export configuration
+                        Contains endpoint, prediction_tags, and confidence_tags mappings
                     - transformed_data (dict[str, Any]): Transformed data to export separately
                         Only processed when path_flag is None
                     - transform_table_name (str): Target table for transformed data export
@@ -87,6 +91,9 @@ class FormatAndExportPrediction:
         transformed_data = input_data.get('transformed_data', None)
         prediction_confidence = input_data['prediction_confidence']
 
+        opc_output_config = input_data.get('opc_output_config', None)
+        pi_web_api_output_config = input_data.get('pi_web_api_output_config', None)
+
         if path_flag is None:
             # Normal prediction path: format prediction data with full metadata
             prediction = await workflow.execute_local_activity_method(
@@ -152,20 +159,36 @@ class FormatAndExportPrediction:
 
             write_transformed_handler = None
 
+        opc_metrics = {}
+
+        # write to pi web api
+        if pi_web_api_output_config:
+            prediction = await workflow.execute_activity_method(
+                Activities.write_pi_web_api_data,
+                {
+                    'pi_web_api_output_config': pi_web_api_output_config,
+                    'data': prediction,
+                    **metadata,
+                },
+                retry_policy=retry_policy,
+                start_to_close_timeout=timedelta(seconds=60),
+            )
+
         # write to opc
-        prediction, opc_metrics = await workflow.execute_activity_method(
+        if opc_output_config:
-            Activities.write_opc_data,
+            prediction, opc_metrics = await workflow.execute_activity_method(
-            {
+                Activities.write_opc_data,
-                'opc_output_config': input_data['opc_output_config'],
+                {
-                'data': prediction,
+                    'opc_output_config': opc_output_config,
-                **metadata,
+                    'data': prediction,
-            },
+                    **metadata,
-            retry_policy=retry_policy,
+                },
-            start_to_close_timeout=timedelta(seconds=60),
+                retry_policy=retry_policy,
-        )
+                start_to_close_timeout=timedelta(seconds=60),
+            )
 
         # write to postgres
-        prediction_handler = workflow.execute_activity_method(
+        await workflow.execute_activity_method(
             Activities.export_data_to_postgres,
             {
                 **metadata,
@@ -178,8 +201,6 @@ class FormatAndExportPrediction:
             start_to_close_timeout=timedelta(seconds=180),
         )
 
-        await prediction_handler
-
         if write_transformed_handler is not None:
             await write_transformed_handler
 

laborious/workflows/sub_workflows/prediction_process.py

@@ -66,7 +66,10 @@ class PredictionProcess:
                     - mlflow_predict_filters (dict): MLFlow prediction filters
                     - model_retention (int): Model retention period in minutes
                     - path_priority (list[str]): Decision path priority configuration
-                    - opc_output_config (dict): OPC server export configuration
+                    - opc_output_config (dict, optional): OPC server export configuration
+                    - pi_web_api_output_config (dict, optional): PI Web API export configuration
+                    - save_transform (bool, optional): Whether to save transformed data (default: True)
+                    - prediction_store_policy (str, optional): Data retention policy (default: 'lts:1')
 
         Returns:
             None: The workflow completes successfully when export workflow finishes
@@ -207,6 +210,7 @@ class PredictionProcess:
                 'model_name': model_name,
                 'model_config': model_config,
                 'opc_output_config': input_data['opc_output_config'],
+                'pi_web_api_output_config': input_data['pi_web_api_output_config'],
                 'schema': input_data['schema'],
                 'table_name': input_data['table_name'],
                 'transform_table_name': input_data['transform_table_name'],
@@ -234,7 +238,17 @@ class PredictionProcess:
         Args:
             data: Input data for processing
             path_flag: Path decision from filter (STOP, CONTINUE, REPEAT)
-            input_data: Complete workflow input configuration
+            input_data: Complete workflow input configuration including:
+                - metadata (dict): Workflow execution metadata
+                - schema (str): Database schema
+                - table_name (str): Target table for predictions
+                - transform_table_name (str): Target table for transformed data
+                - model_id (str): ML model identifier
+                - model_name (str): ML model name
+                - model_config (dict, optional): Model configuration
+                - opc_output_config (dict, optional): OPC server export configuration
+                - pi_web_api_output_config (dict, optional): PI Web API export configuration
+                - prediction_store_policy (str, optional): Data retention policy
             confidence: Confidence level from filter validation
             last_timestamp: Last processed timestamp
             comment: Additional information about the filter result
@@ -244,7 +258,7 @@ class PredictionProcess:
 
         Path Handling:
             - STOP: Terminates workflow execution
-            - CONTINUE: Proceeds with normal processing
+            - CONTINUE: Delegates to FormatAndExportPrediction workflow with current data
             - REPEAT: Repeats last prediction if available
         """
         metadata = input_data['metadata']
@@ -294,6 +308,7 @@ class PredictionProcess:
                     'transform_table_name': transform_table_name,
                     'comment': comment,
                     'opc_output_config': input_data['opc_output_config'],
+                    'pi_web_api_output_config': input_data['pi_web_api_output_config'],
                     'prediction_store_policy': input_data['prediction_store_policy'],
                 },
             )

pyproject.toml

@@ -123,7 +123,7 @@ markers = [
 ]
 
 [tool.coverage.run]
-source = ["model_manager"]
+source = ["laborious"]
 omit = [
     "*/tests/*",
     "*/venv/*",

requirements-dev.txt

@@ -13,7 +13,8 @@ types-requests>=2.31.0   # Type stubs for requests
 pytest>=7.4.0            # Testing framework
 pytest-cov>=4.1.0        # Coverage plugin for pytest
 pytest-asyncio>=0.21.0   # Async test support (already in main requirements)
+testcontainers[postgres] # PostgreSQL containers for E2E tests
 
 # Development Tools
 ipython>=8.12.0          # Enhanced Python shell
-ipdb>=0.13.13            # IPython debugger
+ipdb>=0.13.13            # IPython debugger

requirements-light.txt

@@ -3,7 +3,7 @@ psycopg2-binary
 sqlalchemy
 asyncua
 redis
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.6.1
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.8.0
 prometheus-client
 botocore
 boto3

requirements.txt

@@ -3,8 +3,8 @@ psycopg2-binary
 sqlalchemy
 asyncua
 redis
-git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.6.1
+git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.8.2
-git+ssh://git@github.com/Aignosi/sientia-mlops-library.git@0.40.6
+git+ssh://git@github.com/Aignosi/sientia-mlops-library.git@0.40.7
 prometheus-client
 botocore
 boto3
@@ -12,4 +12,5 @@ s3fs
 pyarrow
 kaleido
 hyperopt
-shap
+shap
+pycurl

sonar-project.properties

@@ -3,7 +3,7 @@ sonar.projectName=sientia-dataops-laborious_temporal
 sonar.sources=laborious
 sonar.tests=tests
 sonar.projectVersion=1.0.0
-sonar.coverage.exclusions=laborious/worker/worker.py
+sonar.coverage.exclusions=laborious/worker/*
 sonar.qualitygate.wait=true
 sonar.qualitygate.timeout=300
 sonar.python.coverage.reportPaths=coverage.xml

tests/laborious/activities/test_activities.py

@@ -3,8 +3,10 @@ from unittest.mock import ANY, AsyncMock, MagicMock, patch
 from pytest import mark
 
 from laborious.activities.activities import Activities
+from laborious.activities.api import API
 from laborious.activities.gates import Gates
 from laborious.activities.mlflow import MLFlow
+from laborious.activities.model_metrics import ModelMetrics
 from laborious.activities.opc import OPC
 from laborious.activities.storage import Storage
 
@@ -13,9 +15,17 @@ from laborious.activities.storage import Storage
 @patch('laborious.activities.activities.MLFlow.__init__')
 @patch('laborious.activities.activities.OPC.__init__')
 @patch('laborious.activities.activities.Gates.__init__')
+@patch('laborious.activities.activities.ModelMetrics.__init__')
+@patch('laborious.activities.activities.API.__init__')
 @patch('laborious.activities.activities.MetricsController')
 def test___init__(
-    mock_metrics_controller, mock_gates_init, mock_opc_init, mock_mlflow_init, mock_storage_init
+    mock_metrics_controller,
+    mock_api_init,
+    mock_model_metrics_init,
+    mock_gates_init,
+    mock_opc_init,
+    mock_mlflow_init,
+    mock_storage_init,
 ):
     postgres_config = {
         'host': 'localhost',
@@ -43,6 +53,12 @@ def test___init__(
         'group_id': 'test-group',
     }
 
+    pi_web_api_config = {
+        'base_url': 'https://test-pi-server.com',
+        'auth_type': 'bearer',
+        'auth_token': 'test_token',
+    }
+
     logger = MagicMock()
     notification_handler = MagicMock()
 
@@ -51,6 +67,7 @@ def test___init__(
         mlflow_config=mlflow_config,
         minio_config=minio_config,
         opc_config=opc_config,
+        pi_web_api_config=pi_web_api_config,
         logger=logger,
         notification_handler=notification_handler,
     )
@@ -60,6 +77,8 @@ def test___init__(
     assert isinstance(activities, MLFlow)
     assert isinstance(activities, OPC)
     assert isinstance(activities, Gates)
+    assert isinstance(activities, ModelMetrics)
+    assert isinstance(activities, API)
 
     mock_storage_init.assert_called_once_with(
         ANY,
@@ -103,13 +122,39 @@ def test___init__(
         metrics_controller=mock_metrics_controller.return_value,
     )
 
+    mock_model_metrics_init.assert_called_once_with(
+        ANY,
+        logger=logger,
+        notification_handler=notification_handler,
+        metrics_controller=mock_metrics_controller.return_value,
+    )
+
+    mock_api_init.assert_called_once_with(
+        ANY,
+        base_url=pi_web_api_config['base_url'],
+        auth_type=pi_web_api_config['auth_type'],
+        auth_token=pi_web_api_config['auth_token'],
+        logger=logger,
+        notification_handler=notification_handler,
+        metrics_controller=mock_metrics_controller.return_value,
+    )
+
 
 @mark.asyncio
 @patch('laborious.activities.activities.Storage')
 @patch('laborious.activities.activities.MLFlow')
 @patch('laborious.activities.activities.OPC')
 @patch('laborious.activities.activities.Gates')
-async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_storage_init):
+@patch('laborious.activities.activities.ModelMetrics')
+@patch('laborious.activities.activities.API')
+async def test_shutdown(
+    mock_api_init,
+    mock_model_metrics_init,
+    mock_gates_init,
+    mock_opc_init,
+    mock_mlflow_init,
+    mock_storage_init,
+):
     mock_opc_init.close = AsyncMock()
     postgres_config = {
         'host': 'localhost',
@@ -137,6 +182,12 @@ async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_s
         'group_id': 'test-group',
     }
 
+    pi_web_api_config = {
+        'base_url': 'https://test-pi-server.com',
+        'auth_type': 'bearer',
+        'auth_token': 'test_token',
+    }
+
     logger = MagicMock()
     notification_handler = MagicMock()
 
@@ -145,6 +196,7 @@ async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_s
         mlflow_config=mlflow_config,
         minio_config=minio_config,
         opc_config=opc_config,
+        pi_web_api_config=pi_web_api_config,
         logger=logger,
         notification_handler=notification_handler,
     )
@@ -154,3 +206,5 @@ async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_s
     mock_storage_init.close.assert_called_once()
     mock_mlflow_init.close.assert_called_once()
     mock_gates_init.close.assert_called_once()
+    mock_model_metrics_init.close.assert_called_once()
+    mock_api_init.close.assert_called_once()

tests/laborious/activities/test_api.py

@@ -0,0 +1,422 @@
+from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
+
+import pytest_asyncio
+from pytest import fixture, mark
+from sientia_do.notifications.models import NotificationLevel
+
+from laborious.activities.api import API, PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+
+metadata = {
+    'metadata': {
+        'model_id': 'test_model',
+        'model_name': 'test_model',
+        'workflow_name': 'test_workflow',
+        'schema_name': 'test_schedule',
+    },
+}
+
+
+def _create_mock_dataframe(to_dict_return=None):
+    """Helper function to create a mocked DataFrame for testing."""
+    mock_df = MagicMock()
+    mock_head = MagicMock()
+
+    def get_column_values(key):
+        if key == 'prediction':
+            return MagicMock(values=[0.75])
+        elif key == 'prediction_confidence':
+            return MagicMock(values=[0.95])
+        else:
+            return MagicMock(values=['2024-01-01T00:00:00+00:00'])
+
+    mock_head.__getitem__.side_effect = get_column_values
+    mock_df.head.return_value = mock_head
+
+    if to_dict_return is None:
+        to_dict_return = {
+            'prediction': [0.75],
+            'prediction_confidence': [0.95],
+            'timestamp': ['2024-01-01T00:00:00+00:00'],
+        }
+    mock_df.to_dict.return_value = to_dict_return
+
+    return mock_df
+
+
+@fixture
+def base_input_data():
+    """Base input data for PI Web API tests."""
+    return {
+        **metadata,
+        'data': {
+            'prediction': [0.75],
+            'prediction_confidence': [0.95],
+            'timestamp': ['2024-01-01T00:00:00+00:00'],
+        },
+        'pi_web_api_output_config': {
+            'endpoint': 'https://test-pi-server.com/piwebapi',
+            'prediction_tags': {'tag1': 'web_id_1'},
+            'confidence_tags': {'tag2': 'web_id_2'},
+        },
+    }
+
+
+def test__init__():
+    api = API(
+        base_url='https://test-pi-server.com',
+        auth_type='bearer',
+        auth_token='test_token',
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+
+    assert api.pi_web_api_client is not None
+
+
+@pytest_asyncio.fixture
+@patch('laborious.activities.api.PIWebAPIClient')
+def api(mock_pi_web_api_client):
+    mock_client = MagicMock()
+    mock_client.write_value = AsyncMock()
+    mock_client.close = MagicMock()
+    mock_client.base_url = 'https://test-pi-server.com'
+    mock_pi_web_api_client.return_value = mock_client
+
+    api_instance = API(
+        base_url='https://test-pi-server.com',
+        auth_type='bearer',
+        auth_token='test_token',
+        logger=MagicMock(),
+        notification_handler=MagicMock(),
+        metrics_controller=AsyncMock(),
+    )
+    api_instance.send_notification_async = AsyncMock()
+    api_instance.info = MagicMock()
+    api_instance.error = MagicMock()
+    api_instance.emit_metric = AsyncMock()
+    api_instance.get_core_labels = MagicMock(
+        return_value={
+            'pod_id': 'test_pod',
+            'model_name': 'test_model',
+            'workflow_name': 'test_workflow',
+        }
+    )
+    return api_instance
+
+
+@mark.asyncio
+@patch('laborious.activities.api.DataFrame')
+async def test_write_pi_web_api_data_success(mock_dataframe, api, base_input_data):
+    input_data = {
+        **base_input_data,
+        'pi_web_api_output_config': {
+            'endpoint': 'https://test-pi-server.com/piwebapi',
+            'prediction_tags': {'tag1': 'web_id_1', 'tag2': 'web_id_2'},
+            'confidence_tags': {'tag3': 'web_id_3', 'tag4': 'web_id_4'},
+        },
+    }
+
+    mock_dataframe.return_value = _create_mock_dataframe()
+
+    # Mock successful responses
+    api.pi_web_api_client.write_value.side_effect = [
+        [{'WebId': 'web_id_1', 'Errors': []}, {'WebId': 'web_id_2', 'Errors': []}],
+        [{'WebId': 'web_id_3', 'Errors': []}, {'WebId': 'web_id_4', 'Errors': []}],
+    ]
+
+    result = await api.write_pi_web_api_data(input_data)
+
+    api.pi_web_api_client.write_value.assert_has_calls(
+        [
+            call(
+                web_ids=['web_id_1', 'web_id_2'],
+                value={
+                    'Timestamp': '2024-01-01T00:00:00+00:00',
+                    'Value': 0.75,
+                },
+                metadata=metadata['metadata'],
+            ),
+            call(
+                web_ids=['web_id_3', 'web_id_4'],
+                value={
+                    'Timestamp': '2024-01-01T00:00:00+00:00',
+                    'Value': 0.95,
+                },
+                metadata=metadata['metadata'],
+            ),
+        ]
+    )
+
+    assert result == {
+        'prediction': [0.75],
+        'prediction_confidence': [0.95],
+        'timestamp': ['2024-01-01T00:00:00+00:00'],
+    }
+
+
+@mark.asyncio
+@patch('laborious.activities.api.DataFrame')
+async def test_write_pi_web_api_data_prediction_error(mock_dataframe, api, base_input_data):
+    mock_dataframe.return_value = _create_mock_dataframe(
+        {
+            'prediction': [0.75],
+            'prediction_confidence': [PI_WEB_API_PREDICTION_ERROR_CONFIDENCE],
+            'timestamp': ['2024-01-01T00:00:00+00:00'],
+        }
+    )
+
+    api.pi_web_api_client.write_value.side_effect = Exception('Prediction write failed')
+
+    result = await api.write_pi_web_api_data(base_input_data)
+
+    api.send_notification_async.assert_called_once_with(
+        metadata=metadata['metadata'],
+        notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
+        message="Error writing prediction data to PI Web API: Prediction write failed\n Tags: {'tag1': 'web_id_1'}",
+        block='write_pi_web_api_data',
+        level=NotificationLevel.ERROR,
+        attachment_content=ANY,
+    )
+
+    assert result['prediction_confidence'][0] == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+    assert api.pi_web_api_client.write_value.call_count == 1
+
+
+@mark.asyncio
+@patch('laborious.activities.api.DataFrame')
+async def test_write_pi_web_api_data_confidence_error(mock_dataframe, api, base_input_data):
+    mock_dataframe.return_value = _create_mock_dataframe()
+
+    # First call succeeds, second fails
+    api.pi_web_api_client.write_value.side_effect = [
+        [{'WebId': 'web_id_1', 'Errors': []}],
+        Exception('Confidence write failed'),
+    ]
+
+    result = await api.write_pi_web_api_data(base_input_data)
+
+    api.send_notification_async.assert_called_once_with(
+        metadata=metadata['metadata'],
+        notification_id='WRITE_PI_WEB_API_CONFIDENCE_ERROR',
+        message="Error writing confidence data to PI Web API: Confidence write failed\n Tags: {'tag2': 'web_id_2'}",
+        block='write_pi_web_api_data',
+        level=NotificationLevel.ERROR,
+        attachment_content=ANY,
+    )
+
+    assert result == {
+        'prediction': [0.75],
+        'prediction_confidence': [0.95],
+        'timestamp': ['2024-01-01T00:00:00+00:00'],
+    }
+    assert api.pi_web_api_client.write_value.call_count == 2
+
+
+@mark.asyncio
+@patch('laborious.activities.api.DataFrame')
+async def test_write_pi_web_api_data_empty_tags(mock_dataframe, api, base_input_data):
+    input_data = {
+        **base_input_data,
+        'pi_web_api_output_config': {
+            'endpoint': 'https://test-pi-server.com/piwebapi',
+            'prediction_tags': {},
+            'confidence_tags': {},
+        },
+    }
+
+    mock_dataframe.return_value = _create_mock_dataframe()
+
+    # Mock empty responses
+    api.pi_web_api_client.write_value.side_effect = [
+        [],
+        [],
+    ]
+
+    result = await api.write_pi_web_api_data(input_data)
+
+    api.pi_web_api_client.write_value.assert_has_calls(
+        [
+            call(
+                web_ids=[],
+                value={
+                    'Timestamp': '2024-01-01T00:00:00+00:00',
+                    'Value': 0.75,
+                },
+                metadata=metadata['metadata'],
+            ),
+            call(
+                web_ids=[],
+                value={
+                    'Timestamp': '2024-01-01T00:00:00+00:00',
+                    'Value': 0.95,
+                },
+                metadata=metadata['metadata'],
+            ),
+        ]
+    )
+
+    assert result == {
+        'prediction': [0.75],
+        'prediction_confidence': [0.95],
+        'timestamp': ['2024-01-01T00:00:00+00:00'],
+    }
+
+
+@mark.asyncio
+async def test_close(api):
+    api.close()
+
+    api.pi_web_api_client.close.assert_called_once()
+
+
+@mark.asyncio
+async def test_process_pi_web_api_response_success(api):
+    """Test successful processing of PI Web API response with all tags written."""
+    response_data = [
+        {'WebId': 'web_id_1', 'Errors': []},
+        {'WebId': 'web_id_2', 'Errors': []},
+    ]
+    tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
+    core_labels = {
+        'pod_id': 'test_pod',
+        'model_name': 'test_model',
+        'workflow_name': 'test_workflow',
+    }
+
+    confidence, message = await api.process_pi_web_api_response(
+        response_data=response_data,
+        tags=tags,
+        core_labels=core_labels,
+        metadata=metadata['metadata'],
+    )
+
+    assert confidence == 0
+    assert message == ''
+    assert api.emit_metric.call_count == 2
+    # Verify that emit_metric was called with correct tags structure
+    call_args_list = api.emit_metric.call_args_list
+    assert len(call_args_list) == 2
+    # Check that all calls include core_labels and tag_name
+    for call_args in call_args_list:
+        assert 'tag_name' in call_args.kwargs['tags']
+        assert call_args.kwargs['tags']['tag_name'] in ['tag1', 'tag2']
+
+
+@mark.asyncio
+async def test_process_pi_web_api_response_with_errors(api):
+    """Test processing response with errors in some tags."""
+    response_data = [
+        {'WebId': 'web_id_1', 'Errors': ['Error writing tag']},
+        {'WebId': 'web_id_2', 'Errors': []},
+    ]
+    tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
+    core_labels = {
+        'pod_id': 'test_pod',
+        'model_name': 'test_model',
+        'workflow_name': 'test_workflow',
+    }
+
+    confidence, message = await api.process_pi_web_api_response(
+        response_data=response_data,
+        tags=tags,
+        core_labels=core_labels,
+        metadata=metadata['metadata'],
+    )
+
+    assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+    assert (
+        message
+        == "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag2'] tags were written."
+    )
+    assert api.emit_metric.call_count == 2
+
+
+@mark.asyncio
+async def test_process_pi_web_api_response_missing_tags(api):
+    """Test processing response when number of written tags doesn't match expected."""
+    response_data = [
+        {'WebId': 'web_id_1', 'Errors': []},
+    ]
+    tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
+    core_labels = {
+        'pod_id': 'test_pod',
+        'model_name': 'test_model',
+        'workflow_name': 'test_workflow',
+    }
+
+    confidence, message = await api.process_pi_web_api_response(
+        response_data=response_data,
+        tags=tags,
+        core_labels=core_labels,
+        metadata=metadata['metadata'],
+    )
+
+    assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+    assert (
+        message
+        == "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag1'] tags were written."
+    )
+    api.send_notification_async.assert_called_once()
+    call_args = api.send_notification_async.call_args
+    assert call_args.kwargs['notification_id'] == 'WRITE_PI_WEB_API_PREDICTION_ERROR'
+    assert call_args.kwargs['level'] == NotificationLevel.ERROR
+
+
+@mark.asyncio
+async def test_process_pi_web_api_response_missing_webid(api):
+    """Test processing response when WebId is missing in response item."""
+    response_data = [
+        {'Errors': []},
+        {'WebId': 'web_id_2', 'Errors': []},
+    ]
+    tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
+    core_labels = {
+        'pod_id': 'test_pod',
+        'model_name': 'test_model',
+        'workflow_name': 'test_workflow',
+    }
+
+    confidence, message = await api.process_pi_web_api_response(
+        response_data=response_data,
+        tags=tags,
+        core_labels=core_labels,
+        metadata=metadata['metadata'],
+    )
+
+    assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+    assert (
+        message
+        == "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag2'] tags were written."
+    )
+    api.error.assert_any_call('The response did not contain some WebIds', metadata['metadata'])
+
+
+@mark.asyncio
+async def test_process_pi_web_api_response_missing_tag_name(api):
+    """Test processing response when tag name is not found for WebId."""
+    response_data = [
+        {'WebId': 'unknown_web_id', 'Errors': []},
+    ]
+    tags = {'tag1': 'web_id_1'}
+    core_labels = {
+        'pod_id': 'test_pod',
+        'model_name': 'test_model',
+        'workflow_name': 'test_workflow',
+    }
+
+    confidence, message = await api.process_pi_web_api_response(
+        response_data=response_data,
+        tags=tags,
+        core_labels=core_labels,
+        metadata=metadata['metadata'],
+    )
+
+    assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
+    assert (
+        message
+        == "The number of written tags does not match the number of tag names: Expected ['tag1'] tags, but [] tags were written."
+    )
+    api.error.assert_any_call(
+        'The response did not contain the tag name for WebId unknown_web_id', metadata['metadata']
+    )

tests/laborious/utils/repository/test_model_repository.py

@@ -566,6 +566,12 @@ async def test_download_model_transform(mlflow_repository):
     assert result == (mlflow_repository.load_transform_model.return_value, None)
 
 
+def test_detect_and_parse_datetime_index_empty(mlflow_repository):
+    input_data = DataFrame()
+    response = mlflow_repository.detect_and_parse_datetime_index(input_data, metadata['metadata'])
+    assert response.equals(input_data)
+
+
 invalid_cases = [
     (
         {'value': {'2024-01-01 12:00:00': 1, 2024: 2}},
@@ -812,7 +818,14 @@ async def test_fit_models_not_df_target_name_none_and_not_in_model(
     data = MagicMock()
 
     output = await mlflow_repository.fit_models(
-        'model_name', data, 'latest_production_id', metadata['metadata'], 'sklearn', 'pyfunc', None
+        'model_name',
+        data,
+        'latest_production_id',
+        metadata['metadata'],
+        'sklearn',
+        False,
+        'pyfunc',
+        None,
     )
 
     mlflow_repository.download_model.assert_has_calls(
@@ -908,6 +921,7 @@ async def test_fit_models_df_target_name_not_none_and_in_model(
         'latest_production_id',
         metadata['metadata'],
         'sklearn',
+        False,
         'pyfunc',
         'feat_1',
     )
@@ -1425,6 +1439,7 @@ async def test_retrain_model(mlflow_repository):
         model_name=model_name,
         data=data,
         transform_flavor='sklearn',
+        skip_transform=False,
         predict_flavor='pyfunc',
         target_name='target',
         metadata=metadata['metadata'],

tests/laborious/utils/test_connectors_config.py

@@ -3,9 +3,7 @@ from os import environ
 from laborious.utils.connectors_config import (
     build_minio_config,
     build_mlflow_config,
-    build_mongodb_config,
     build_opc_config,
-    build_postgres_config,
 )
 
 
@@ -92,79 +90,6 @@ def test_build_opc_config_with_defaults():
     assert config['1']['reconnection_interval'] == 120
 
 
-def test_build_postgres_config_with_env_vars():
-    # Arrange
-    environ['POSTGRES_HOST'] = 'test-host'
-    environ['POSTGRES_PORT'] = '5433'
-    environ['POSTGRES_USER'] = 'test-user'
-    environ['POSTGRES_PASSWORD'] = 'test-pass'
-    environ['POSTGRES_DBNAME'] = 'test-db'
-    environ['POSTGRES_MIN_CONNECTIONS'] = '10'
-    environ['POSTGRES_MAX_CONNECTIONS'] = '30'
-
-    # Act
-    config = build_postgres_config()
-
-    # Assert
-    assert config['host'] == 'test-host'
-    assert config['port'] == 5433
-    assert config['user'] == 'test-user'
-    assert config['password'] == 'test-pass'
-    assert config['dbname'] == 'test-db'
-    assert config['min_connections'] == 10
-    assert config['max_connections'] == 30
-
-
-def test_build_postgres_config_with_defaults():
-    # Arrange
-    environ.pop('POSTGRES_HOST', None)
-    environ.pop('POSTGRES_PORT', None)
-    environ.pop('POSTGRES_USER', None)
-    environ.pop('POSTGRES_PASSWORD', None)
-    environ.pop('POSTGRES_DBNAME', None)
-    environ.pop('POSTGRES_MIN_CONNECTIONS', None)
-    environ.pop('POSTGRES_MAX_CONNECTIONS', None)
-
-    # Act
-    config = build_postgres_config()
-
-    # Assert
-    assert config['host'] == 'localhost'
-    assert config['port'] == 5432
-    assert config['user'] == 'sientia'
-    assert config['password'] == 'sientia'
-    assert config['dbname'] == 'sientia'
-    assert config['min_connections'] == 5
-    assert config['max_connections'] == 20
-
-
-def test_build_mongo_db_config_with_env_vars():
-    environ['MONGODB_USERNAME'] = 'sientia1'
-    environ['MONGODB_PASSWORD'] = 'sientia1'
-    environ['MONGODB_URL'] = 'localhost:27018'
-    environ['MONGODB_DATABASE_NAME'] = 'test_db'
-    environ['MONGODB_TTL_INDEX_HOURS'] = '1'
-
-    assert build_mongodb_config() == {
-        'connection_string': 'mongodb://sientia1:sientia1@localhost:27018',
-        'database_name': 'test_db',
-        'ttl_index_seconds': 3600,
-    }
-
-
-def test_build_mongo_db_config_with_defaults():
-    environ.pop('MONGODB_USERNAME', None)
-    environ.pop('MONGODB_PASSWORD', None)
-    environ.pop('MONGODB_DATABASE_NAME', None)
-    environ.pop('MONGODB_URL', None)
-    environ.pop('MONGODB_TTL_INDEX_HOURS', None)
-    assert build_mongodb_config() == {
-        'connection_string': 'mongodb://root:wKZDbMNU1c@localhost:27018',
-        'database_name': 'sientia',
-        'ttl_index_seconds': 3600,
-    }
-
-
 def test_build_minio_config_with_env_vars():
     environ['MINIO_ENDPOINT_URL'] = 'http://test-host'
     environ['MINIO_ACCESS_KEY'] = 'test-key'

tests/laborious/workflows/subworkflows/test_format_and_export_prediction.py

@@ -375,3 +375,283 @@ async def test_run_default_path_flag(workflow_mock, format_and_export_prediction
 
     assert workflow_mock.execute_activity_method.call_count == 3
     assert workflow_mock.execute_local_activity_method.call_count == 1
+
+
+@mark.asyncio
+@patch(
+    'laborious.workflows.sub_workflows.format_and_export_prediction.workflow',
+    new_callable=AsyncMock,
+)
+async def test_run_none_path_flag_with_pi_web_api(workflow_mock, format_and_export_prediction):
+    input_data = {
+        'metadata': metadata,
+        'path_flag': None,
+        'data': {'test': 'data'},
+        'timestamp': '2021-01-01',
+        'model_id': 1,
+        'prediction_confidence': 0,
+        'schema': 'test_schema',
+        'table_name': 'test_table',
+        'pi_web_api_output_config': {
+            'endpoint': 'https://test-pi-server.com',
+            'prediction_tags': {},
+            'confidence_tags': {},
+        },
+        'prediction_store_policy': 'erl:1',
+    }
+
+    pi_web_api_data = MagicMock()
+
+    workflow_mock.execute_activity_method.side_effect = [
+        pi_web_api_data,  # write_pi_web_api_data
+        MagicMock(),  # export_data_to_postgres
+        MagicMock(),  # write_metrics
+    ]
+
+    await format_and_export_prediction.run(input_data)
+
+    workflow_mock.execute_local_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.format_prediction,
+                {
+                    'data': input_data['data'],
+                    'timestamp': input_data['timestamp'],
+                    'model_id': input_data['model_id'],
+                    'prediction_confidence': input_data['prediction_confidence'],
+                    'prediction_store_policy': input_data['prediction_store_policy'],
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.write_pi_web_api_data,
+                {
+                    'pi_web_api_output_config': input_data['pi_web_api_output_config'],
+                    'data': workflow_mock.execute_local_activity_method.return_value,
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.export_data_to_postgres,
+                {
+                    'schema': input_data['schema'],
+                    'table_name': input_data['table_name'],
+                    'data': pi_web_api_data,
+                    'timestamp_conversion': {
+                        'column': 'timestamp',
+                        'format': DATETIME_FORMAT_WITH_TZ,
+                    },
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.write_metrics,
+                {
+                    **metadata,
+                    'prediction': pi_web_api_data,
+                    'opc_metrics': {},
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    assert workflow_mock.execute_activity_method.call_count == 3
+    assert workflow_mock.execute_local_activity_method.call_count == 1
+
+
+@mark.asyncio
+@patch(
+    'laborious.workflows.sub_workflows.format_and_export_prediction.workflow',
+    new_callable=AsyncMock,
+)
+async def test_run_none_path_flag_with_pi_web_api_and_opc(
+    workflow_mock, format_and_export_prediction
+):
+    input_data = {
+        'metadata': metadata,
+        'path_flag': None,
+        'data': {'test': 'data'},
+        'timestamp': '2021-01-01',
+        'model_id': 1,
+        'prediction_confidence': 0,
+        'schema': 'test_schema',
+        'table_name': 'test_table',
+        'opc_output_config': {'test': 'config'},
+        'pi_web_api_output_config': {
+            'endpoint': 'https://test-pi-server.com',
+            'prediction_tags': {},
+            'confidence_tags': {},
+        },
+        'prediction_store_policy': 'erl:1',
+    }
+
+    prediction_data = MagicMock()
+    pi_web_api_data = MagicMock()
+    opc_metrics = MagicMock()
+
+    workflow_mock.execute_activity_method.side_effect = [
+        pi_web_api_data,  # write_pi_web_api_data
+        (prediction_data, opc_metrics),  # write_opc_data
+        MagicMock(),  # export_data_to_postgres
+        MagicMock(),  # write_metrics
+    ]
+
+    await format_and_export_prediction.run(input_data)
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.write_pi_web_api_data,
+                {
+                    'pi_web_api_output_config': input_data['pi_web_api_output_config'],
+                    'data': workflow_mock.execute_local_activity_method.return_value,
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            ),
+            call(
+                Activities.write_opc_data,
+                {
+                    'opc_output_config': input_data['opc_output_config'],
+                    'data': pi_web_api_data,
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            ),
+        ]
+    )
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.export_data_to_postgres,
+                {
+                    'schema': input_data['schema'],
+                    'table_name': input_data['table_name'],
+                    'data': prediction_data,
+                    'timestamp_conversion': {
+                        'column': 'timestamp',
+                        'format': DATETIME_FORMAT_WITH_TZ,
+                    },
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.write_metrics,
+                {
+                    **metadata,
+                    'prediction': prediction_data,
+                    'opc_metrics': opc_metrics,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    assert workflow_mock.execute_activity_method.call_count == 4
+    assert workflow_mock.execute_local_activity_method.call_count == 1
+
+
+@mark.asyncio
+@patch(
+    'laborious.workflows.sub_workflows.format_and_export_prediction.workflow',
+    new_callable=AsyncMock,
+)
+async def test_run_default_path_flag_with_pi_web_api(workflow_mock, format_and_export_prediction):
+    input_data = {
+        'metadata': metadata,
+        'path_flag': 'default',
+        'data': {'test': 'data'},
+        'timestamp': '2021-01-01',
+        'model_id': 1,
+        'prediction_confidence': 0,
+        'schema': 'test_schema',
+        'table_name': 'test_table',
+        'pi_web_api_output_config': {
+            'endpoint': 'https://test-pi-server.com',
+            'prediction_tags': {},
+            'confidence_tags': {},
+        },
+        'comment': 'test_comment',
+    }
+
+    pi_web_api_data = MagicMock()
+
+    workflow_mock.execute_activity_method.side_effect = [
+        pi_web_api_data,  # write_pi_web_api_data
+        MagicMock(),  # export_data_to_postgres
+        MagicMock(),  # write_metrics
+    ]
+
+    await format_and_export_prediction.run(input_data)
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.write_pi_web_api_data,
+                {
+                    'pi_web_api_output_config': input_data['pi_web_api_output_config'],
+                    'data': workflow_mock.execute_local_activity_method.return_value,
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    workflow_mock.execute_activity_method.assert_has_calls(
+        [
+            call(
+                Activities.export_data_to_postgres,
+                {
+                    'schema': input_data['schema'],
+                    'table_name': input_data['table_name'],
+                    'data': pi_web_api_data,
+                    'timestamp_conversion': {
+                        'column': 'timestamp',
+                        'format': DATETIME_FORMAT_WITH_TZ,
+                    },
+                    **metadata,
+                },
+                retry_policy=ANY,
+                start_to_close_timeout=ANY,
+            )
+        ]
+    )
+
+    assert workflow_mock.execute_activity_method.call_count == 3
+    assert workflow_mock.execute_local_activity_method.call_count == 1

tests/laborious/workflows/subworkflows/test_prediction_process.py

@@ -40,6 +40,7 @@ async def test_run(workflow_mock, prediction_process):
         'model_config': {'retention': '30'},
         'path_priority': ['continue', 'repeat', 'stop'],
         'opc_output_config': {'test': 'config'},
+        'pi_web_api_output_config': {'test': 'config'},
         'prediction_store_policy': 'lts:1',
     }
 
@@ -183,6 +184,7 @@ async def test_run(workflow_mock, prediction_process):
             'model_name': 'test_model_name',
             'model_config': input_data['model_config'],
             'opc_output_config': input_data['opc_output_config'],
+            'pi_web_api_output_config': input_data['pi_web_api_output_config'],
             'schema': input_data['schema'],
             'table_name': input_data['table_name'],
             'transform_table_name': input_data['transform_table_name'],
@@ -729,6 +731,11 @@ async def test_path_flag_handler_continue(workflow_mock, prediction_process):
             'model_name': model_name,
             'model_config': model_config,
             'opc_output_config': {'test': 'config'},
+            'pi_web_api_output_config': {
+                'endpoint': 'https://test-pi-server.com',
+                'prediction_tags': {},
+                'confidence_tags': {},
+            },
             'prediction_store_policy': prediction_store_policy,
         },
         confidence,
@@ -755,6 +762,11 @@ async def test_path_flag_handler_continue(workflow_mock, prediction_process):
             'transform_table_name': 'test_transform_table',
             'comment': 'Prediction Process',
             'opc_output_config': {'test': 'config'},
+            'pi_web_api_output_config': {
+                'endpoint': 'https://test-pi-server.com',
+                'prediction_tags': {},
+                'confidence_tags': {},
+            },
             'prediction_store_policy': prediction_store_policy,
         },
     )
@@ -788,6 +800,11 @@ async def test_path_flag_handler_unknown(workflow_mock, prediction_process):
             'model_name': model_name,
             'model_config': model_config,
             'opc_output_config': {'test': 'config'},
+            'pi_web_api_output_config': {
+                'endpoint': 'https://test-pi-server.com',
+                'prediction_tags': {},
+                'confidence_tags': {},
+            },
             'prediction_store_policy': prediction_store_policy,
         },
         confidence,

tests/laborious/workflows/test_predictions_batch.py

@@ -34,6 +34,7 @@ async def test_run(workflow_mock: AsyncMock, predictions_batch: PredictionsBatch
         'table_name': 'test_table',
         'transform_table_name': 'test_transform_table',
         'opc_output_config': 'test_opc_output_config',
+        'pi_web_api_output_config': 'test_pi_web_api_output_config',
         'datetime_columns': ['timestamp', 'created_at'],
         'prediction_store_policy': 'erl:1',
         'model_config': {'retention': '30'},
@@ -73,7 +74,9 @@ async def test_run(workflow_mock: AsyncMock, predictions_batch: PredictionsBatch
         'model_config': input_data.get('model_config', {}),
         'path_priority': input_data.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT']),
         'opc_output_config': input_data.get('opc_output_config', {}),
+        'pi_web_api_output_config': input_data.get('pi_web_api_output_config', {}),
         'prediction_store_policy': input_data.get('prediction_store_policy', 'lts:1'),
+        'save_transform': input_data.get('save_transform', True),
     }
 
     workflow_mock.execute_child_workflow.assert_has_calls(

validate.sh

@@ -70,14 +70,14 @@ FAILED_STEPS=()
 # Step 1: Code Formatting Check (Ruff)
 # - default: check only
 # - --fix: write changes
-if ! run_step "1. Code Formatting (Ruff)" "if \$FIX_MODE; then ruff format laborious/ tests/; else ruff format --check laborious/ tests/; fi"; then
+if ! run_step "1. Code Formatting (Ruff)" "if \$FIX_MODE; then ruff format laborious/ tests/; else ruff format --check laborious/ tests/ e2e/; fi"; then
     FAILED_STEPS+=("Code Formatting")
 fi
 
 # Step 2: Linting (Ruff)
 # - default: check only
 # - --fix: apply autofixes
-if ! run_step "2. Code Linting (Ruff)" "if \$FIX_MODE; then ruff check --fix laborious/ tests/; else ruff check laborious/ tests/; fi"; then
+if ! run_step "2. Code Linting (Ruff)" "if \$FIX_MODE; then ruff check --fix laborious/ tests/; else ruff check laborious/ tests/ e2e/; fi"; then
     FAILED_STEPS+=("Linting")
 fi
 

values.yaml

@@ -3,17 +3,17 @@
 # Declare variables to be passed into your templates.
 
 # This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
-replicaCount: 2
+replicaCount: 1
 
 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
 image:
-  repository: aignosi.azurecr.io/sientia-module-courier
+  repository: aignosi.azurecr.io/sientia-module
   # This sets the pull policy for images.
   pullPolicy: Always
   # Overrides the image tag whose default is the chart appVersion.
   tag: "1.1.2"
 
-0# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
+# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
 imagePullSecrets:
 - name: docker-hub-secret
 # This is to override the chart name.
@@ -62,24 +62,32 @@ resources:
     cpu: 1000m  # 1 CPU core
     memory: 2Gi  # 2 GB memory
 
+# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
 # This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
 livenessProbe:
   exec:
     command:
       - sh
       - -c
-      - pgrep -f "laborious.worker.worker"
+      - |
-  initialDelaySeconds: 20
+        curl -sf http://localhost:9090/metrics | grep -q '^app_up{.*} 1'
-  periodSeconds: 30
+  initialDelaySeconds: 660
+  periodSeconds: 15
+  timeoutSeconds: 5
+  failureThreshold: 3
 
 readinessProbe:
   exec:
     command:
       - sh
       - -c
-      - pgrep -f "laborious.worker.worker"
+      - |
-  initialDelaySeconds: 10
+        curl -sf http://localhost:9090/metrics | grep -q '^app_up{.*} 1'
-  periodSeconds: 15
+  initialDelaySeconds: 600
+  periodSeconds: 10
+  timeoutSeconds: 3
+  failureThreshold: 2
+
 
 
 # This section is for setting up autoscaling more information can be found here: https://kubernetes.io/docs/concepts/workloads/autoscaling/
@@ -149,7 +157,7 @@ env:
   - name: GITHUB_REPO_URL
     value: "git@github.com:Aignosi/sientia-dataops-laborious_temporal.git"
   - name: GITHUB_BRANCH
-    value: "feature/SIENTIAPDE-1273"
+    value: "feature/SIENTIAPDE-1478"
   - name: PYTHON_APP
     value: "laborious.worker.worker"
 
@@ -187,8 +195,6 @@ env:
   - name: OPC_URL
     value: "opc.tcp://sientia-opc-simulator-opc.sientia.svc.cluster.local:4840"
 
-  - name: KAFKA_BOOTSTRAP_SERVERS
-    value: "kafka.kafka.svc.cluster.local:9092"
 
   - name: LOG_LEVEL
     value: "DEBUG"
@@ -226,6 +232,19 @@ env:
   - name: MINIO_DEFAULT_BUCKET
     value: "sientia"
 
+  - name: PI_WEB_API_BASE_URL
+    value: "https://pivision.votorantimcimentos.com/piwebapi"
+  - name: PI_WEB_API_AUTH_TYPE
+    value: "basic"
+  - name: PI_WEB_API_AUTH_TOKEN
+    valueFrom:
+      secretKeyRef:
+        name: pi-web-api-auth-token
+        key: token
+
+  - name: PYPI_SERVER
+    value: "http://library-distribution-server.library.svc.cluster.local:5000"
+
 ssh:
   enabled: true
   secretName: git-ssh-key-sientia-laborious-worker