gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #35 from Aignosi/feature/SIENTIAPDE-1478

Browse Source 
Feature: PI Web API Integration, Enhanced PredictionsBatch E2E Tests, and System Refinements
This commit is contained in:
vitor-aignosi
2026-01-23 13:48:03 -03:00
committed by GitHub
parent d2e5e8ce93 ab5e91fa07
commit bda479af5f
37 changed files with 4822 additions and 833 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/release.yml vendored
View File

@@ -21,5 +21,5 @@ jobs:
permissions: write-all
with:
project_name: 'laborious'
version: ${{ github.event.inputs.version || '' }}
release_version: ${{ github.event.inputs.version || '' }}
secrets: inherit

88
README.md
View File

@@ -1,6 +1,6 @@
# Sientia DataOps Laborious
A comprehensive, Temporal-based ML orchestration system for industrial data processing and model inference. Laborious delivers enterprise-grade batch prediction, model management, optional real-time export (OPC), and automated retraining with strong data quality validation and observability.
A comprehensive, Temporal-based ML orchestration system for industrial data processing and model inference. Laborious delivers enterprise-grade batch prediction, model management, optional real-time export (OPC and PI Web API), and automated retraining with strong data quality validation and observability.
## 📑 Table of Contents
@@ -71,7 +71,7 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
- **Temporal Workflow Orchestration**: Robust workflow management with retries and fault tolerance
- **Data Quality Gates**: Configurable filtering for input data and MLFlow API responses
- **Multi-Model Support**: Flexible model management with retention and versioning
- **Optional Real-time Export**: PostgreSQL persistence and OPC server integration for industrial systems
- **Optional Real-time Export**: PostgreSQL persistence, OPC server integration, and PI Web API integration for industrial systems
- **Comprehensive Monitoring**: Prometheus metrics and structured logging for observability
### Advanced Capabilities
@@ -140,6 +140,9 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
- Model retraining and production updates
- Reference data retrieval from MLflow Model Registry
- `opc.py`: OPC UA export to industrial systems (optional)
- `api.py`: PI Web API export operations (optional)
- Prediction and confidence data writing to PI Web API
- Error handling and notification integration
- `activities.py`: Aggregates activity interfaces
#### **Data Services (`laborious/utils/`)**
@@ -154,7 +157,7 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
```
Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform →
MLFlow Prediction → Response Validation → Format & Export
├─→ Predictions → PostgreSQL [+ OPC]
├─→ Predictions → PostgreSQL [+ OPC] [+ PI Web API]
└─→ Transformed Data → PostgreSQL (optional)
```
@@ -170,6 +173,7 @@ Production Update → Notification & Monitoring
- **MLFlow API Authentication**: Username/password
- **Database Security**: Encrypted connections and credential management
- **OPC Certificates** (if enabled): Client/server certs
- **PI Web API Authentication**: Bearer token or basic authentication
- **Kubernetes Secrets**: Secure secret storage
#### **Network Security**
@@ -230,6 +234,11 @@ The **PredictionsBatch** workflow is the main entry point for batch prediction p
"opc_output_config": {
"server_id": "opc_server_1",
"tags": ["prediction_output"]
},
"pi_web_api_output_config": {
"endpoint": "https://pi-server.com/piwebapi",
"prediction_tags": {"tag1": "web_id_1"},
"confidence_tags": {"tag2": "web_id_2"}
}
}
```
@@ -299,7 +308,12 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M
},
"model_retention": 60,
"path_priority": ["STOP", "CONTINUE", "REPEAT"],
"opc_output_config": {...}
"opc_output_config": {...},
"pi_web_api_output_config": {
"endpoint": "https://pi-server.com/piwebapi",
"prediction_tags": {"tag1": "web_id_1"},
"confidence_tags": {"tag2": "web_id_2"}
}
}
```
@@ -322,19 +336,21 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an
- **Data Formatting**: Formats prediction data for different output destinations
- **PostgreSQL Export**: Persists predictions to database with metrics
- **OPC Integration**: Writes predictions to OPC servers for real-time access
- **PI Web API Integration**: Writes predictions and confidence to PI Web API for industrial systems
- **Metrics Recording**: Tracks export operations and performance metrics
#### Execution Flow
1. **Path Decision**: Determines formatting path based on configuration
2. **Data Formatting**: Formats prediction data for specific output requirements
3. **Transformed Data Processing**: Optionally formats and exports transformed data separately
4. **PostgreSQL Export**: Writes formatted predictions to database
5. **OPC Export**: Writes predictions to OPC servers
6. **Metrics Recording**: Records export performance and success metrics
4. **PI Web API Export**: Writes predictions and confidence to PI Web API (if configured)
5. **OPC Export**: Writes predictions to OPC servers (if configured)
6. **PostgreSQL Export**: Writes formatted predictions to database
7. **Metrics Recording**: Records export performance and success metrics
#### Key Features
- **Flexible Formatting**: Configurable output formats for different destinations
- **Multi-Destination Export**: PostgreSQL and OPC server integration
- **Multi-Destination Export**: PostgreSQL, OPC server, and PI Web API integration
- **Transformed Data Export**: Optional separate export of MLFlow transformed data
- **Performance Monitoring**: Comprehensive metrics for export operations
- **Error Handling**: Robust error handling with notification integration
@@ -342,13 +358,14 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an
#### Architecture Diagram
```mermaid
flowchart LR
A[1. format_prediction/format_default_prediction] --> B[2. format_transformed_data] --> C[3. write_opc_data] --> D[4. export_data_to_postgres] --> E[5. write_metrics]
A[1. format_prediction/format_default_prediction] --> B[2. format_transformed_data] --> C[3. write_pi_web_api_data] --> D[4. write_opc_data] --> E[5. export_data_to_postgres] --> F[6. write_metrics]
A -.-> Format[Data Formatting]
B -.-> Transform[Transformed Data]
C -.-> OPC[OPC Servers]
D -.-> PostgreSQL[(PostgreSQL)]
E -.-> Prometheus[Prometheus]
C -.-> PIWebAPI[PI Web API]
D -.-> OPC[OPC Servers]
E -.-> PostgreSQL[(PostgreSQL)]
F -.-> Prometheus[Prometheus]
```
#### Transformed Data Export
@@ -395,6 +412,7 @@ flowchart LR
- MinIO object storage (for MLFlow artifacts)
- MongoDB server (for notifications)
- OPC server(s) if using OPC export
- PI Web API server if using PI Web API export
**Note**: External dependencies must be available either through:
- Kubernetes cluster deployment
@@ -685,6 +703,9 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
| `OPC_PRIVATE_KEY_PATH` | OPC private key path | `None` | No |
| `OPC_SERVER_CERT_PATH` | OPC server certificate path | `None` | No |
| `OPC_RECONNECTION_INTERVAL` | OPC reconnection interval (ms) | `120` | No |
| `PI_WEB_API_BASE_URL` | PI Web API server base URL | `None` | No |
| `PI_WEB_API_AUTH_TYPE` | PI Web API authentication type (basic/bearer) | `None` | No |
| `PI_WEB_API_AUTH_TOKEN` | PI Web API authentication token | `None` | No |
| `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
| `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
| `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
@@ -735,6 +756,37 @@ For single OPC server, use individual environment variables:
- `OPC_SERVER_CERT_PATH`
- `OPC_RECONNECTION_INTERVAL`
### PI Web API Configuration
PI Web API configuration is built from environment variables using the `build_api_config` function from `sientia_do.connectors_config`. The configuration includes:
- `PI_WEB_API_BASE_URL`: Base URL of the PI Web API server
- `PI_WEB_API_AUTH_TYPE`: Authentication type ('basic' or 'bearer')
- `PI_WEB_API_AUTH_TOKEN`: Authentication token for API access
The PI Web API export is optional and can be configured per workflow through the `pi_web_api_output_config` parameter:
```json
{
"pi_web_api_output_config": {
"endpoint": "https://pi-server.com/piwebapi",
"prediction_tags": {
"tag1": "web_id_1",
"tag2": "web_id_2"
},
"confidence_tags": {
"tag3": "web_id_3",
"tag4": "web_id_4"
}
}
}
```
Where:
- `endpoint`: PI Web API endpoint URL
- `prediction_tags`: Dictionary mapping tag names to web IDs for prediction values
- `confidence_tags`: Dictionary mapping tag names to web IDs for confidence values
### Workflow Configuration
MongoDB pipeline configuration:
@@ -833,7 +885,8 @@ laborious/
│ ├── activities.py # Main activities orchestrator
│ ├── gates.py # Data quality gates and filtering
│ ├── mlflow.py # MLFlow model operations
│ └── opc.py # OPC server operations
│ ├── opc.py # OPC server operations
│ └── api.py # PI Web API operations
├── workflows/ # Temporal workflow definitions
│ ├── predictions_batch.py # Main batch prediction workflow
│ ├── minimal_retrain.py # Model retraining workflow
@@ -886,7 +939,14 @@ laborious/
- Check certificate and key file paths
- Review OPC server logs for connection issues
5. **Workflow Execution Failures**
5. **PI Web API Connection Failures**
- Verify PI Web API server is accessible
- Check authentication credentials and token validity
- Verify web IDs exist and have write permissions
- Review PI Web API server logs for connection issues
- Check notification system for error details
6. **Workflow Execution Failures**
- Review activity error logs and notifications
- Check data quality filter configurations
- Verify input data format and required fields

3
e2e/__init__.py Normal file
View File

@@ -0,0 +1,3 @@
"""
End-to-end tests for laborious temporal workflows.
"""

452
e2e/conftest.py Normal file
View File

@@ -0,0 +1,452 @@
"""
Pytest configuration and fixtures for E2E tests.
"""
from unittest.mock import AsyncMock, MagicMock, patch
import pandas as pd
import pytest
import pytest_asyncio
from sqlalchemy import create_engine, text
from testcontainers.postgres import PostgresContainer
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
from laborious.activities.activities import Activities
from laborious.workflows.predictions_batch import PredictionsBatch
from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
from laborious.workflows.sub_workflows.format_and_export_prediction import FormatAndExportPrediction
from sientia_do.notifications.handlers import CoreNotificationHandler
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
# Test constants
TEST_MONGODB_CONNECTION_STRING = 'mongodb://localhost:27017'
TEST_DATABASE_NAME = 'test_db'
@pytest_asyncio.fixture(scope='session')
def postgres_container():
"""
Create a PostgreSQL container using testcontainers.
This fixture creates a real PostgreSQL database in a Docker container
that will be used for all tests in the session.
"""
postgres = PostgresContainer('postgres:15')
postgres.start()
yield postgres
postgres.stop()
@pytest_asyncio.fixture
def postgres_engine(postgres_container):
"""
Create SQLAlchemy engine for PostgreSQL test database.
This fixture creates a connection to the PostgreSQL container
created by the postgres_container fixture.
"""
engine = create_engine(postgres_container.get_connection_url())
yield engine
engine.dispose()
def _create_schema_and_tables(engine):
"""
Helper function to create schema and tables in the given engine.
Creates predictions_schema with:
- laborious_data: Input data table for queries
- predictions: Output predictions table
- transformed_data: Output transformed data table
"""
# Use begin() to ensure transaction is properly committed
with engine.begin() as conn:
# Create predictions_schema
conn.execute(text("CREATE SCHEMA IF NOT EXISTS predictions_schema"))
# Create laborious_data table (input data from sensors)
create_laborious_data_sql = """
CREATE TABLE IF NOT EXISTS predictions_schema.laborious_data (
id SERIAL NOT NULL,
model_id int4 NOT NULL,
variable text NOT NULL,
value numeric NULL,
"timestamp" timestamptz NOT NULL,
created_at timestamptz NOT NULL,
PRIMARY KEY (id)
);
"""
conn.execute(text(create_laborious_data_sql))
# Create predictions table
create_predictions_sql = """
CREATE TABLE if not exists predictions_schema.predictions (
id SERIAL NOT NULL ,
model_id int4 NOT NULL,
prediction numeric NULL,
prediction_confidence numeric NOT NULL,
response_time numeric NOT NULL,
prediction_status text NOT NULL,
"timestamp" timestamptz NOT NULL,
created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
"comments" text NULL,
PRIMARY KEY (id, created_at)
);
"""
conn.execute(text(create_predictions_sql))
# Create transformed_data table
create_transformed_sql = """
CREATE TABLE IF NOT EXISTS predictions_schema.transformed_data (
id SERIAL NOT NULL,
model_id int4 NOT NULL,
variable text NOT NULL,
value numeric NULL,
"timestamp" timestamptz NOT NULL,
created_at timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
PRIMARY KEY (id)
);
"""
conn.execute(text(create_transformed_sql))
@pytest_asyncio.fixture(autouse=True)
def setup_postgres_schema_and_tables(postgres_engine):
"""
Automatically create necessary schema and tables before each test.
This fixture runs automatically (autouse=True) and ensures
that the predictions_schema and tables exist with the correct structure.
"""
_create_schema_and_tables(postgres_engine)
yield
@pytest_asyncio.fixture
def mock_logger():
"""Mock logger for testing."""
def message(message):
print(f"[LOG] {message}")
def custom_message(message, _metadata={}):
print(f"[LOG] {message}")
logger = MagicMock()
logger.info = MagicMock(
side_effect=message
)
logger.debug = MagicMock(
side_effect=message
)
logger.error = MagicMock(
side_effect=message
)
logger.warning = MagicMock(
side_effect=message
)
logger.custom_info = MagicMock(
side_effect=custom_message
)
logger.custom_debug = MagicMock(
side_effect=custom_message
)
logger.custom_error = MagicMock(
side_effect=custom_message
)
logger.custom_warning = MagicMock(
side_effect=custom_message
)
return logger
@pytest_asyncio.fixture
def mock_mongo_client():
"""
Mock MongoDB client to avoid real connections.
This fixture mocks the pymongo.MongoClient used by CoreNotificationHandler,
allowing us to use a real NotificationHandler instance without connecting to MongoDB.
"""
mock_client = MagicMock()
mock_db = MagicMock()
mock_collection = MagicMock()
# Configure the mock chain: client[database] -> db[collection] -> collection
mock_client.__getitem__.return_value = mock_db
mock_db.__getitem__.return_value = mock_collection
# Mock server_info() to avoid connection attempts
mock_client.server_info = MagicMock()
# Mock insert_one for notifications
mock_collection.insert_one = MagicMock()
return mock_client
@pytest_asyncio.fixture
def notification_handler(mock_logger, mock_mongo_client):
"""
Create a real NotificationHandler instance with mocked MongoDB client.
This fixture creates a real CoreNotificationHandler instance but mocks
the underlying MongoDB connection to avoid real database connections.
"""
# Patch MongoClient where it's imported in the handlers module
with patch('sientia_do.notifications.handlers.MongoClient', return_value=mock_mongo_client):
handler = CoreNotificationHandler(
connection_string=TEST_MONGODB_CONNECTION_STRING,
database=TEST_DATABASE_NAME,
logger=mock_logger,
project_name='laborious',
)
yield handler
handler.shutdown()
@pytest_asyncio.fixture
def metrics_controller(mock_logger):
"""Create a real MetricsController instance."""
return MetricsController(logger=mock_logger)
@pytest_asyncio.fixture
def mock_minio_repository():
"""Mock MinIO repository for object storage operations."""
mock_repo = MagicMock()
# Mock repository methods
mock_repo.put_parquet_from_dataframe = AsyncMock(return_value='test-object-key')
mock_repo.get_parquet_as_dataframe = AsyncMock(return_value=pd.DataFrame())
mock_repo.minio_bucket = 'test-bucket'
return mock_repo
@pytest_asyncio.fixture
def mock_pi_web_api_repository():
"""Mock PI Web API repository for PI Web API operations."""
mock_repo = MagicMock()
mock_repo.write_value = AsyncMock(
return_value={
'Items': [
{
'WebId': 'web_id_1'
}
]
}
)
mock_repo.close = MagicMock()
return mock_repo
@pytest_asyncio.fixture
def mock_opc_repository():
"""Mock OPC repository for OPC operations."""
mock_repo = MagicMock()
mock_repo.write_data = AsyncMock(
return_value=(True, {'response_time': 0.1})
)
mock_repo.disconnect = MagicMock()
return mock_repo
@pytest_asyncio.fixture
def patch_create_engine(postgres_engine):
"""Patch create_engine to return test postgres_engine."""
with patch('sientia_do.temporal.activities.postgres.create_engine', return_value=postgres_engine):
yield
@pytest_asyncio.fixture
def patch_minio_repository(mock_minio_repository):
"""Patch MinioRepository to return mock."""
with patch('laborious.utils.repository.minio_repository.MinioRepository', return_value=mock_minio_repository):
yield
@pytest_asyncio.fixture
def patch_pi_web_api_repository(mock_pi_web_api_repository):
"""Patch MLflowRepository to return mock."""
with patch('laborious.activities.api.PIWebAPIClient', return_value=mock_pi_web_api_repository):
yield
@pytest_asyncio.fixture
def mock_mlflow_models():
"""Create mock models for MLflow load_model methods."""
# Mock transform model - returns DataFrame with same index as input
mock_transform_model = MagicMock()
def mock_transform_predict(data):
num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
print(data.to_csv())
print(data.index)
result = pd.DataFrame({
'feature_1': [0.234] * num_rows,
'feature_2': [0.783] * num_rows,
})
result.index = data.index
return result
mock_transform_model.predict = MagicMock(side_effect=mock_transform_predict)
# Mock predict model - returns array/list of predictions
mock_predict_model = MagicMock()
def mock_predict_predict(data):
num_rows = max(len(data), 1) if hasattr(data, '__len__') else 1
return [0.5] * num_rows
mock_predict_model.predict = MagicMock(side_effect=mock_predict_predict)
# Mock PyFuncModel for compressed models
mock_pyfunc_model = MagicMock()
mock_pyfunc_model._model_impl = MagicMock()
mock_pyfunc_model._model_impl.python_model = mock_transform_model
return {
'transform_model': mock_transform_model,
'predict_model': mock_predict_model,
'pyfunc_model': mock_pyfunc_model,
}
@pytest_asyncio.fixture
def patch_mlflow(mock_mlflow_models):
"""Patch mlflow module in repository with load_model mocks."""
mock_mlflow = MagicMock()
# Mock sklearn.load_model
def mock_sklearn_load_model(model_uri):
if 'data_model' in model_uri or 'transform' in model_uri.lower():
return mock_mlflow_models['transform_model']
return mock_mlflow_models['predict_model']
mock_mlflow.sklearn = MagicMock()
mock_mlflow.sklearn.load_model = MagicMock(side_effect=mock_sklearn_load_model)
# Mock pyfunc.load_model
def mock_pyfunc_load_model(model_uri):
if 'artifacts' in model_uri or 'tmp' in model_uri:
return mock_mlflow_models['pyfunc_model']
if 'data_model' in model_uri or 'transform' in model_uri.lower():
return mock_mlflow_models['transform_model']
return mock_mlflow_models['predict_model']
mock_mlflow.pyfunc = MagicMock()
mock_mlflow.pyfunc.load_model = MagicMock(side_effect=mock_pyfunc_load_model)
# Mock pytorch.load_model
mock_mlflow.pytorch = MagicMock()
mock_mlflow.pytorch.load_model = MagicMock(return_value=mock_mlflow_models['predict_model'])
# Mock other mlflow methods that might be called
mock_mlflow.set_tracking_uri = MagicMock()
mock_mlflow.get_run = MagicMock(return_value=MagicMock(info=MagicMock(artifact_uri='mlflow-artifacts:/test_run_id')))
mock_mlflow.tracking = MagicMock()
mock_mlflow.tracking.MlflowClient = MagicMock(return_value=MagicMock(
search_registered_models=MagicMock(return_value=[MagicMock(name='test_model')]),
search_model_versions=MagicMock(return_value=[MagicMock(
current_stage='Production',
version='1',
source='runs:/artifacts/test_run_id'
)])
))
with patch('laborious.utils.repository.model_repository.mlflow', new=mock_mlflow):
yield mock_mlflow
@pytest_asyncio.fixture(scope='function')
async def test_activities(
postgres_engine,
postgres_container,
mock_logger,
notification_handler,
metrics_controller,
mock_minio_repository,
patch_create_engine,
patch_minio_repository,
patch_mlflow,
patch_pi_web_api_repository,
mock_opc_repository
):
"""
Create Activities instance with test dependencies.
This fixture creates a real Activities instance with:
- PostgreSQL database (via testcontainers)
- Mocked MinIO client
- Real NotificationHandler and MetricsController (with mocked underlying services)
"""
activities = Activities(
postgres_config={
'host': 'localhost',
'port': postgres_container.get_exposed_port(5432),
'user': 'test',
'password': 'test',
'dbname': 'test',
'min_connections': 1,
'max_connections': 5,
},
mlflow_config={
'host': 'http://localhost',
'port': '5000',
'username': 'test',
'password': 'test',
},
minio_config={
'endpoint_url': 'http://localhost:9000',
'access_key': 'test',
'secret_key': 'test',
'region_name': 'us-east-1',
'default_bucket': 'test-bucket',
},
opc_config={},
pi_web_api_config={
'base_url': 'http://localhost:8080',
'auth_type': 'bearer',
'auth_token': 'test_token',
},
logger=mock_logger,
notification_handler=notification_handler,
)
activities.opc_repository = {
'1': mock_opc_repository,
}
try:
yield activities
finally:
# Cleanup - ALWAYS runs, even if test fails
await activities.shutdown()
@pytest_asyncio.fixture(scope='function')
async def temporal_test_env():
"""Create Temporal test environment."""
env = await WorkflowEnvironment.start_time_skipping()
async with env:
yield env
@pytest_asyncio.fixture(scope='function')
async def temporal_worker(temporal_test_env, test_activities):
"""Create Temporal worker with test activities."""
async with Worker(
temporal_test_env.client,
task_queue='test-queue',
workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
activities=[
test_activities.load_custom_query,
test_activities.get_last_timestamp,
test_activities.input_gate,
test_activities.request_transform,
test_activities.mlflow_response_gate,
test_activities.mlflow_content_gate,
test_activities.request_predict,
test_activities.repeat_last_prediction,
test_activities.format_prediction,
test_activities.format_transformed_data,
test_activities.format_default_prediction,
test_activities.write_pi_web_api_data,
test_activities.write_opc_data,
test_activities.export_data_to_postgres,
test_activities.write_metrics,
],
) as worker:
yield worker

509
e2e/scenarios.md Normal file
View File

@@ -0,0 +1,509 @@
# Test Scenarios for Predictions Batch Workflow
This document describes all possible test scenarios for the `predictions_batch` workflow and its child workflows `prediction_process` and `format_and_export_prediction`.
## Workflow Overview
The `predictions_batch` workflow:
1. Loads data using a custom SQL query
2. Prepares prediction configuration
3. Delegates to `prediction_process` child workflow which:
- Retrieves last timestamp for incremental processing
- Applies input data quality gates
- Executes MLFlow transform operation
- Validates transform response
- Executes MLFlow predict operation
- Validates predict response
- Delegates to `format_and_export_prediction` child workflow
4. The `format_and_export_prediction` workflow:
- Formats prediction data (normal or default)
- Exports to PI Web API (optional)
- Exports to OPC server (optional)
- Exports to PostgreSQL
- Writes metrics
---
## 1. Predictions Batch - Main Workflow Scenarios
### 1.1 Success Scenarios
#### Scenario 1.1.1: Happy Path - Complete Success
**Description**: Workflow completes successfully with valid SQL query and all activities succeed
**Input**:
- Valid `schedule_name`, `model_name`, `model_id`
- Valid `query` returning non-empty DataFrame
- Valid `schema`, `table_name`, `transform_table_name`
- Optional `datetime_columns` for timestamp parsing
- Optional `input_filters`, `mlflow_transform_filters`, `mlflow_predict_filters`
- Optional `path_priority`, `opc_output_config`, `pi_web_api_output_config`
**Expected Behavior**:
- `load_custom_query` returns DataFrame with data
- Workflow prepares prediction input with all configurations
- `prediction_process` child workflow executes successfully
- All gates pass with no issues
- Transform and predict operations succeed
- Data exported to PostgreSQL
- Metrics written
**Assertions**:
- SQL query executed once
- `prediction_process` workflow called with correct parameters
- Data exists in PostgreSQL (predictions table)
- Metrics recorded
- No errors raised
---
### 1.2 Error Scenarios
#### Scenario 1.2.1: SQL Query Execution Error
**Description**: SQL query fails due to syntax error or connection issue
**Input**:
- Invalid SQL query (syntax error)
- Or database connection unavailable
**Expected Behavior**:
- `load_custom_query` raises exception (caught by Temporal retry policy)
- Notification sent with SQL error details
- After retries, activity may return empty data or workflow may fail
- If empty data returned, workflow completes with early exit via input gate
**Assertions**:
- Error notification sent
- Workflow completes (either fails or exits early)
- No data in predictions table
---
#### Scenario 1.2.2: Missing Required Parameters
**Description**: Essential parameters missing from input
**Input**:
- Missing `query` or `model_id` or `schema` or `table_name`
**Expected Behavior**:
- Workflow or activity raises KeyError or validation error
- Workflow fails immediately
**Assertions**:
- Workflow fails with parameter error
- Error notification sent
- No child workflow called
---
#### Scenario 1.2.3: Invalid Datetime Column Specification
**Description**: Datetime column specified doesn't exist in query results
**Input**:
- `datetime_columns: ['nonexistent_column']`
- Query results don't have this column
**Expected Behavior**:
- `load_custom_query` may raise KeyError or warning
- Depending on implementation, workflow may fail or continue
- Error notification sent
**Assertions**:
- Error raised or warning logged
- Workflow behavior depends on error handling policy
---
## 2. Prediction Process - Child Workflow Scenarios
### 2.1 Input gate Early Exit Scenarios
#### Scenario 2.1.1: Input Gate Triggers CONTINUE
**Description**: Input gate determines data should use previous prediction
**Input**:
- Data that should continue with input data as prediction
- `input_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE
**Expected Behavior**:
- `input_gate` returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with input data directly
- MLFlow transform and predict skipped
- Data exported as-is
**Assertions**:
- `input_gate` called
- MLFlow operations NOT called
- Export workflow called with original data
- Workflow completes
#### Scenario 2.1.2: Input Gate Triggers STOP
**Description**: Input data quality gate fails with STOP policy
**Input**:
- Data with EMPTY_DATA or other critical issues
- `input_filters` configured with `POLICY: 'STOP'`
**Expected Behavior**:
- `input_gate` returns `path_flag='STOP'`
- `path_flag_handler` detects STOP
- Workflow returns early without calling MLFlow
- No prediction exported
**Assertions**:
- `input_gate` called
- `path_flag_handler` returns True (early exit)
- MLFlow transform NOT called
- Export workflow NOT called
- Workflow completes without error
#### Scenario 2.1.3: Input Gate Triggers REPEAT
**Description**: Input gate determines data should repeat last prediction
**Input**:
- Data with quality issues that require using previous prediction
- `input_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT
**Expected Behavior**:
- `input_gate` returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- MLFlow transform and predict skipped
- Last prediction repeated and exported
**Assertions**:
- `input_gate` called
- MLFlow operations NOT called
- `repeat_last_prediction` activity called
- Workflow completes
---
### 2.2 Transform gate Early Exit Scenarios
#### Scenario 2.2.1: Transform Gate Triggers CONTINUE
**Description**: Transform response gate determines data should continue despite issues
**Input**:
- Valid input data
- Transform response has quality issues but policy is CONTINUE
- `mlflow_transform_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE
**Expected Behavior**:
- `request_transform` succeeds
- `mlflow_response_gate` for transform returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with transform data
- MLFlow predict skipped
- Transform data exported as-is
**Assertions**:
- Transform completed
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- Export workflow called with transform data
- Workflow completes
---
#### Scenario 2.2.2: Transform Gate Triggers STOP
**Description**: Transform response validation fails with STOP policy
**Input**:
- Valid input data
- Transform response has critical errors
- `mlflow_transform_filters` configured with `POLICY: 'STOP'`
**Expected Behavior**:
- `request_transform` succeeds but response invalid
- `mlflow_response_gate` for transform returns `path_flag='STOP'`
- Workflow exits without calling predict or export
**Assertions**:
- Transform completed but validation failed
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- Export workflow NOT called
- Workflow completes without error
---
#### Scenario 2.2.3: Transform Gate Triggers REPEAT
**Description**: Transform response gate determines data should repeat last prediction
**Input**:
- Valid input data
- Transform response has quality issues that require using previous prediction
- `mlflow_transform_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT
**Expected Behavior**:
- `request_transform` succeeds but response has issues
- `mlflow_response_gate` for transform returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- MLFlow predict skipped
- Last prediction repeated and exported
**Assertions**:
- Transform completed but validation triggered REPEAT
- `mlflow_response_gate` called for transform
- MLFlow predict NOT called
- `repeat_last_prediction` activity called
- Workflow completes
---
### 2.3 Predict gate Early Exit Scenarios
#### Scenario 2.3.1: Predict Gate Triggers CONTINUE
**Description**: Predict response gate determines data should continue despite issues
**Input**:
- Valid input and transform data
- Predict response has quality issues but policy is CONTINUE
- `mlflow_predict_filters` configured with `POLICY: 'CONTINUE'`
- `path_priority` includes CONTINUE
**Expected Behavior**:
- `request_predict` succeeds
- `mlflow_response_gate` for predict returns `path_flag='CONTINUE'`
- `path_flag_handler` calls export workflow with predict data
- Prediction exported despite quality issues
**Assertions**:
- Transform and predict completed
- `mlflow_response_gate` called for predict
- Export workflow called with predict data
- Workflow completes
---
#### Scenario 2.3.2: Predict Gate Triggers STOP
**Description**: Prediction validation fails with STOP policy
**Input**:
- Valid input and transform
- Predict response has critical errors
- `mlflow_predict_filters` configured with `POLICY: 'STOP'`
**Expected Behavior**:
- `request_predict` succeeds but response invalid
- `mlflow_response_gate` for predict returns `path_flag='STOP'`
- Workflow exits without export
**Assertions**:
- Transform completed
- Predict completed but validation failed
- Export workflow NOT called
- Workflow completes without error
---
#### Scenario 2.3.3: Predict Gate Triggers REPEAT
**Description**: Predict response gate determines data should repeat last prediction
**Input**:
- Valid input and transform data
- Predict response has quality issues that require using previous prediction
- `mlflow_predict_filters` configured with `POLICY: 'REPEAT'`
- `path_priority` includes REPEAT
**Expected Behavior**:
- `request_predict` succeeds but response has issues
- `mlflow_response_gate` for predict returns `path_flag='REPEAT'`
- `path_flag_handler` calls `repeat_last_prediction` activity
- Last prediction repeated and exported
**Assertions**:
- Transform and predict completed but validation triggered REPEAT
- `mlflow_response_gate` called for predict
- `repeat_last_prediction` activity called
- Export workflow NOT called with current prediction
- Workflow completes
---
## 3. Format and Export Prediction - Child Workflow Scenarios
### 3.1 Success Scenarios
#### Scenario 3.1.1: Default Prediction Export
**Description**: Error prediction path creates default prediction
**Input**:
- `path_flag: 'ERROR'` or other non-None value (not STOP/CONTINUE/REPEAT)
- `comment` provided with error details
**Expected Behavior**:
- `format_default_prediction` called instead of `format_prediction`
- Default prediction created with error metadata
- Exported to PostgreSQL only
- Transformed data NOT processed
- Metrics written
**Assertions**:
- `format_default_prediction` called
- `format_prediction` NOT called
- `format_transformed_data` NOT called
- One PostgreSQL export only
- Default values in prediction data
- Comment included
---
#### Scenario 3.1.2: Export with OPC only
**Description**: Export to PostgreSQL and OPC server only (no PI Web API)
**Input**:
- `path_flag: None`
- `opc_output_config` configured with valid OPC settings
- `pi_web_api_output_config: None` or `{}`
**Expected Behavior**:
- Normal formatting
- PostgreSQL export executed
- OPC export executed
- PI Web API activity skipped
- Metrics written with OPC metrics
**Assertions**:
- PI Web API activity NOT called
- OPC activity called
- PostgreSQL export called
- Metrics written with `opc_metrics` populated
---
#### Scenario 3.1.3: Export with PI Web API only
**Description**: Export to PostgreSQL and PI Web API only (no OPC)
**Input**:
- `path_flag: None`
- `pi_web_api_output_config` configured with valid PI Web API settings
- `opc_output_config: None` or `{}`
**Expected Behavior**:
- Normal formatting
- PostgreSQL export executed
- PI Web API export executed
- OPC activity skipped
- Metrics written without OPC metrics
**Assertions**:
- OPC activity NOT called
- PI Web API activity called
- PostgreSQL export called
- Metrics written with empty `opc_metrics`
---
#### Scenario 3.1.4: Export Without Optional Outputs
**Description**: Export only to PostgreSQL (no OPC or PI Web API)
**Input**:
- `path_flag: None`
- `opc_output_config: None` or `{}`
- `pi_web_api_output_config: None` or `{}`
**Expected Behavior**:
- Normal formatting
- Only PostgreSQL export executed
- OPC and PI Web API activities skipped
- Metrics written without OPC metrics
**Assertions**:
- PI Web API activity NOT called
- OPC activity NOT called
- PostgreSQL export called
- Metrics written with empty `opc_metrics`
---
#### Scenario 3.1.5: Export Without Transformed Data
**Description**: Only prediction exported, no transform table
**Input**:
- `path_flag: None`
- `transformed_data: None` or `save_transform: False`
- `opc_output_config: None` or `{}`
- `pi_web_api_output_config: None` or `{}`
**Expected Behavior**:
- Only prediction formatted and exported
- Transform export skipped
- Single PostgreSQL write
**Assertions**:
- `format_transformed_data` NOT called
- One PostgreSQL export
- Transform table remains empty
---
### 3.2 Error Scenarios
#### Scenario 3.2.1: PI Web API Write Error
**Description**: PI Web API export fails
**Input**:
- Valid prediction
- PI Web API service unavailable or invalid config
**Expected Behavior**:
- `write_pi_web_api_data` raises exception
- Notification sent
- Workflow fails after retries
- PostgreSQL export may not execute (depends on execution order)
**Assertions**:
- PI Web API error notification sent
- Workflow fails
- May impact subsequent exports
---
#### Scenario 3.2.2: OPC Write Error
**Description**: OPC server write fails
**Input**:
- Valid prediction
- OPC server unavailable or invalid configuration
**Expected Behavior**:
- `write_opc_data` raises exception
- Notification sent
- Workflow fails after retries
**Assertions**:
- OPC error notification sent
- Workflow fails
- PostgreSQL export may not execute
---
#### Scenario 3.2.3: PI Web API Partial Write Error
**Description**: Two prediction tags attempt to be written to PI Web API, but only one succeeds
**Input**:
- Valid prediction
- Two prediction tags configured
- PI Web API returns partial success (one tag succeeds, one fails)
**Expected Behavior**:
- `write_pi_web_api_data` processes response
- `process_pi_web_api_response` detects partial failure
- Error confidence set (13)
- Notification sent for failed tag
- Workflow completes with error confidence
**Assertions**:
- One tag written successfully
- One tag failed
- Error confidence set in prediction
- Error notification sent
- Workflow completes
---

829
e2e/test_predictions_batch_format_export.py Normal file
View File

@@ -0,0 +1,829 @@
"""
End-to-end tests for PredictionsBatch workflow - Format and Export scenarios.
"""
import asyncio
from datetime import datetime
from unittest.mock import ANY, AsyncMock, patch, call
import pandas as pd
import pytest
from sientia_do.notifications.models import NotificationLevel
from sqlalchemy import text
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
from laborious.activities.activities import Activities
from laborious.workflows.predictions_batch import PredictionsBatch
base_input_data = {
'schedule_name': 'test-schedule',
'model_name': 'test_model',
'model_id': 301,
'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 301',
'schema': 'predictions_schema',
'table_name': 'predictions',
'transform_table_name': 'transformed_data',
'input_filters': {
'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
},
'mlflow_transform_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'mlflow_predict_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
'opc_output_config': {},
'pi_web_api_output_config': {},
'save_transform': True,
'prediction_store_policy': 'lts:1',
'model_config': {
'retention_minutes': 0,
'transform_flavor': 'sklearn',
'predict_flavor': 'sklearn',
},
'datetime_columns': ['timestamp', 'created_at'],
}
base_query = "SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = {model_id}"
def get_base_input_data(model_id):
return {
**base_input_data,
'model_id': model_id,
'query': base_query.format(model_id=model_id),
}
def insert_sample_data(postgres_engine, model_id, values: list):
with postgres_engine.begin() as conn:
conn.execute(text(f"DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}"))
values_sql = []
for i, value in enumerate(values):
values_sql.append(f"""
({model_id}, 'sensor_{i+1}', {value}, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
""")
insert_sql = f"""
INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
VALUES
{', '.join(values_sql)}
"""
conn.execute(text(insert_sql))
async def start_and_await_workflow(client, input_data, workflow_id):
handle = await client.start_workflow(
PredictionsBatch.run,
input_data,
id=workflow_id,
task_queue='test-queue',
)
print("[TEST] ✓ Workflow started")
print("\n[TEST] 3. Waiting for workflow completion...")
try:
await asyncio.wait_for(handle.result(), timeout=60.0)
print("[TEST] ✓ Workflow completed successfully")
except asyncio.TimeoutError:
pytest.fail("Workflow execution timed out after 60 seconds")
def assert_prediction(
postgres_engine, model_id, prediction: float = 0.5,
prediction_confidence: int = 0, prediction_status: str = 'Good',
comments: str = '',
):
"""
Verify prediction was created with correct values in database
Args:
postgres_engine: Database engine
model_id: Model ID to check
prediction: Expected prediction value (default 0.5 from mock)
prediction_confidence: Expected confidence value (default 0 for normal predictions)
prediction_status: Expected status (default 'Good')
comments: Expected comments (default empty string)
"""
print("\n[TEST] 4. Verifying prediction was created with correct values...")
with postgres_engine.connect() as conn:
result_query = conn.execute(
text(f"SELECT model_id, prediction, prediction_confidence, prediction_status, comments FROM predictions_schema.predictions WHERE model_id = {model_id}")
)
prediction_rows = result_query.fetchall()
assert len(prediction_rows) == 1, f"Expected one prediction record, got {len(prediction_rows)}"
row = prediction_rows[0]
assert row[0] == model_id, f"Expected model_id={model_id}, got {row[0]}"
assert row[1] == prediction, f"Expected prediction={prediction}, got {row[1]}"
assert row[2] == prediction_confidence, f"Expected prediction_confidence={prediction_confidence}, got {row[2]}"
assert row[3] == prediction_status, f"Expected prediction_status='{prediction_status}', got {row[3]}"
assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_1_1_default_prediction_export(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.1.1: Default Prediction Export
Description:
Error prediction path creates default prediction.
Expected Behavior:
- format_default_prediction called instead of format_prediction
- Default prediction created with error metadata
- Exported to PostgreSQL only
- Transformed data NOT processed
- Metrics written
Assertions:
- format_default_prediction called
- format_prediction NOT called
- format_transformed_data NOT called
- One PostgreSQL export only
- Default values in prediction data
- Comment included
"""
client = temporal_test_env.client
model_id = 311
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['pi_web_api_output_config'] = {
'endpoint': 'test_endpoint',
'prediction_tags': {'tag_1': 'web_id_1'},
'confidence_tags': {'tag_2': 'web_id_2'},
}
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
print("\n[TEST] 2. Starting workflow that should create default prediction...")
workflow_id = f'test-default-prediction-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
test_activities.pi_web_api_client.write_value.assert_has_calls(
[
call(
web_ids=['web_id_1'],
value={
'Timestamp': '2024-01-01 12:00:00+0000',
'Value': 0.5,
},
endpoint='test_endpoint',
metadata={
'model_id': 311,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
call(
web_ids=['web_id_2'],
value={
'Timestamp': '2024-01-01 12:00:00+0000',
'Value': 0,
},
endpoint='test_endpoint',
metadata={
'model_id': 311,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
],
any_order=True,
)
test_activities.opc_repository['1'].write_data.assert_has_calls(
[
call('addr_1', 0.5, 'float', ANY,
{
'model_id': 311,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call('addr_2', 0, 'float', ANY,
{
'model_id': 311,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
]
)
assert_prediction(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_1_2_export_with_opc_only(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.1.2: Export with OPC only
Description:
Export to PostgreSQL and OPC server only (no PI Web API).
Expected Behavior:
- Normal formatting
- PostgreSQL export executed
- OPC export executed
- PI Web API activity skipped
- Metrics written with OPC metrics
Assertions:
- PI Web API activity NOT called
- OPC activity called
- PostgreSQL export called
- Metrics written with opc_metrics populated
"""
client = temporal_test_env.client
model_id = 312
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
input_data['pi_web_api_output_config'] = None # No PI Web API config
print("\n[TEST] 2. Starting workflow with OPC only...")
workflow_id = f'test-opc-only-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
test_activities.opc_repository['1'].write_data.assert_has_calls(
[
call('addr_1', 0.5, 'float', ANY,
{
'model_id': 312,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call('addr_2', 0, 'float', ANY,
{
'model_id': 312,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
]
)
test_activities.pi_web_api_client.write_value.assert_not_called()
assert_prediction(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_1_3_export_with_pi_web_api_only(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.1.3: Export with PI Web API only
Description:
Export to PostgreSQL and PI Web API only (no OPC).
Expected Behavior:
- Normal formatting
- PostgreSQL export executed
- PI Web API export executed
- OPC activity skipped
- Metrics written without OPC metrics
Assertions:
- OPC activity NOT called
- PI Web API activity called
- PostgreSQL export called
- Metrics written with empty opc_metrics
"""
client = temporal_test_env.client
model_id = 313
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['pi_web_api_output_config'] = {
'endpoint': 'test_endpoint',
'prediction_tags': {'tag_1': 'web_id_1'},
'confidence_tags': {'tag_2': 'web_id_2'},
}
input_data['opc_output_config'] = None # No OPC config
print("\n[TEST] 2. Starting workflow with PI Web API only...")
workflow_id = f'test-pi-api-only-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
test_activities.pi_web_api_client.write_value.assert_has_calls(
[
call(
web_ids=['web_id_1'],
value={
'Timestamp': '2024-01-01 12:00:00+0000',
'Value': 0.5,
},
endpoint='test_endpoint',
metadata={
'model_id': 313,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
call(
web_ids=['web_id_2'],
value={
'Timestamp': '2024-01-01 12:00:00+0000',
'Value': 0,
},
endpoint='test_endpoint',
metadata={
'model_id': 313,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
],
any_order=True,
)
test_activities.opc_repository['1'].write_data.assert_not_called()
assert_prediction(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_1_4_export_without_optional_outputs(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.1.4: Export Without Optional Outputs
Description:
Export only to PostgreSQL (no OPC or PI Web API).
Expected Behavior:
- Normal formatting
- Only PostgreSQL export executed
- OPC and PI Web API activities skipped
- Metrics written without OPC metrics
Assertions:
- PI Web API activity NOT called
- OPC activity NOT called
- PostgreSQL export called
- Metrics written with empty opc_metrics
"""
client = temporal_test_env.client
model_id = 314
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['opc_output_config'] = None # No OPC config
input_data['pi_web_api_output_config'] = None # No PI Web API config
print("\n[TEST] 2. Starting workflow without optional outputs...")
workflow_id = f'test-no-optional-outputs-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
test_activities.pi_web_api_client.write_value.assert_not_called()
test_activities.opc_repository['1'].write_data.assert_not_called()
assert_prediction(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_1_5_export_without_transformed_data(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.1.5: Export Without Transformed Data
Description:
Only prediction exported, no transform table.
Expected Behavior:
- Only prediction formatted and exported
- Transform export skipped
- Single PostgreSQL write
Assertions:
- format_transformed_data NOT called
- One PostgreSQL export
- Transform table remains empty
"""
client = temporal_test_env.client
model_id = 315
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
with postgres_engine.begin() as conn:
conn.execute(text(f"DELETE FROM predictions_schema.transformed_data WHERE model_id = {model_id}"))
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['save_transform'] = False # Don't save transformed data
input_data['pi_web_api_output_config'] = {
'endpoint': 'test_endpoint',
'prediction_tags': {'tag_1': 'web_id_1'},
'confidence_tags': {'tag_2': 'web_id_2'},
}
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
print("\n[TEST] 2. Starting workflow without transformed data export...")
workflow_id = f'test-no-transform-export-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
test_activities.pi_web_api_client.write_value.assert_has_calls(
[
call(
web_ids=['web_id_1'],
value={
'Timestamp': '2024-01-01 12:00:00+0000',
'Value': 0.5,
},
endpoint='test_endpoint',
metadata={
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
call(
web_ids=['web_id_2'],
value={
'Timestamp': '2024-01-01 12:00:00+0000',
'Value': 0,
},
endpoint='test_endpoint',
metadata={
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
},
),
],
any_order=True,
)
test_activities.opc_repository['1'].write_data.assert_has_calls(
[
call('addr_1', 0.5, 'float', ANY,
{
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
call('addr_2', 0, 'float', ANY,
{
'model_id': 315,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}),
]
)
with postgres_engine.connect() as conn:
result_query = conn.execute(
text(f"SELECT COUNT(*) FROM predictions_schema.transformed_data WHERE model_id = {model_id}")
)
count = result_query.scalar()
assert count == 0, f"Expected transform table to be empty, but found {count} records"
assert_prediction(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_2_1_pi_web_api_write_error(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.2.1: PI Web API Write Error
Description:
PI Web API export fails.
Expected Behavior:
- write_pi_web_api_data raises exception
- Notification sent
- Workflow fails after retries
- PostgreSQL export may not execute (depends on execution order)
Assertions:
- PI Web API error notification sent
- Workflow fails
- May impact subsequent exports
"""
client = temporal_test_env.client
model_id = 321
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
test_activities.pi_web_api_client.write_value.side_effect = Exception(
"PI Web API service unavailable")
input_data = get_base_input_data(model_id)
input_data['pi_web_api_output_config'] = {
'endpoint': 'test_endpoint',
'prediction_tags': {'tag_1': 'web_id_1'},
'confidence_tags': {'tag_2': 'web_id_2'},
}
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
print("\n[TEST] 2. Starting workflow that should fail on PI Web API write...")
workflow_id = f'test-pi-api-error-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_prediction(
postgres_engine, model_id,
prediction_confidence=13,
comments='PI Web API service unavailable',
)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_2_2_opc_write_error(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.2.2: OPC Write Error
Description:
OPC server write fails.
Expected Behavior:
- write_opc_data raises exception
- Notification sent
- Workflow fails after retries
Assertions:
- OPC error notification sent
- Workflow fails
- PostgreSQL export may not execute
"""
client = temporal_test_env.client
model_id = 322
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
test_activities.opc_repository['1'].write_data.return_value = (False, {
'notification_id': 'OPC_WRITE_DATA_ERROR_1',
'message': 'OPC server unavailable',
'block': 'opc_repository',
'level': NotificationLevel.ERROR,
'attachment_content': 'OPC server unavailable',
})
input_data = get_base_input_data(model_id)
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
input_data['pi_web_api_output_config'] = {
'endpoint': 'test_endpoint',
'prediction_tags': {'tag_1': 'web_id_1'},
'confidence_tags': {'tag_2': 'web_id_2'},
}
print("\n[TEST] 2. Starting workflow that should fail on OPC write...")
workflow_id = f'test-opc-error-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_prediction(
postgres_engine, model_id,
prediction_confidence=12,
comments='Some data could not be written to OPC servers',
)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_3_2_3_pi_web_api_partial_write_error(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 3.2.3: PI Web API Partial Write Error
Description:
Two prediction tags attempt to be written to PI Web API, but only one succeeds.
Expected Behavior:
- write_pi_web_api_data processes response
- process_pi_web_api_response detects partial failure
- Error confidence set (13)
- Notification sent for failed tag
- Workflow completes with error confidence
Assertions:
- One tag written successfully
- One tag failed
- Error confidence set in prediction
- Error notification sent
- Workflow completes
"""
client = temporal_test_env.client
model_id = 323
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
test_activities.pi_web_api_client.write_value = AsyncMock(side_effect=[
{
'Items': [
{
'WebId': 'web_id_1',
'Errors': [],
},
]
},
Exception('Tag write failed'),
{
'Items': [
{
'WebId': 'web_id_2',
'Errors': [],
},
]
},
])
input_data = get_base_input_data(model_id)
input_data['pi_web_api_output_config'] = {
'endpoint': 'test_endpoint',
'prediction_tags': {'tag_1': 'web_id_1', 'tag_3': 'web_id_3'},
'confidence_tags': {'tag_2': 'web_id_2'},
}
input_data['opc_output_config'] = {
'1': {
'prediction_tags': {
'addr_1': {
'data_type': 'float',
}
},
'confidence_tags': {
'addr_2': {
'data_type': 'float',
}
},
}
}
print("\n[TEST] 2. Starting workflow with partial PI Web API write error...")
workflow_id = f'test-pi-api-partial-error-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_prediction(
postgres_engine, model_id,
prediction_confidence=13,
comments="The number of written tags does not match the number of tag names: Expected ['tag_1', 'tag_3'] tags, but ['tag_1'] tags were written.",
)
print("\n[TEST] ✓ All assertions passed!")

450
e2e/test_predictions_batch_main_workflow.py Normal file
View File

@@ -0,0 +1,450 @@
"""
End-to-end tests for PredictionsBatch workflow - Main workflow scenarios.
"""
import asyncio
from datetime import datetime
import pandas as pd
import pytest
from sqlalchemy import text
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
from laborious.activities.activities import Activities
from laborious.workflows.predictions_batch import PredictionsBatch
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_1_1_1_happy_path_complete_success(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 1.1.1: Happy Path - Complete Success
Description:
Workflow completes successfully with valid SQL query and all activities succeed.
Process Flow:
1. load_custom_query returns DataFrame with sensor data
2. Workflow prepares prediction input with all configurations
3. prediction_process child workflow executes:
- get_last_timestamp retrieves last processing timestamp
- input_gate validates data quality (passes)
- request_transform calls MLFlow transform (mocked, returns features)
- mlflow_response_gate validates transform response (passes)
- mlflow_content_gate validates transform content (passes)
- request_predict calls MLFlow predict (mocked, returns predictions)
- mlflow_response_gate validates predict response (passes)
- mlflow_content_gate validates predict content (passes)
4. format_and_export_prediction child workflow executes:
- format_prediction formats the prediction data
- format_transformed_data formats transformed data (if save_transform=True)
- export_data_to_postgres saves to database
- write_metrics records execution metrics
Expected Behavior:
- All activities execute successfully without errors
- All gates pass with no quality issues
- Transform and predict operations succeed (mocked)
- Data exported to PostgreSQL predictions table
- Transformed data exported to transformed_data table (if save_transform=True)
- Metrics written successfully
Assertions:
- Workflow completes without raising exceptions
- Data exists in PostgreSQL predictions table with correct model_id
- Data exists in transformed_data table (if save_transform=True)
- Prediction data has expected structure (jsonb with predictions)
- All required fields are populated (model_id, model_name, timestamp, etc)
"""
client = temporal_test_env.client
print("\n[TEST] 1. Inserting test data into PostgreSQL...")
# Insert test data directly into PostgreSQL
# The load_custom_query activity will fetch this data with a real SQL query
with postgres_engine.begin() as conn:
# Clear any existing data for this model_id
conn.execute(text("DELETE FROM predictions_schema.laborious_data WHERE model_id = 123"))
# Insert sensor data that the workflow will query
insert_sql = """
INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
VALUES
(123, 'sensor_1', 23.5, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00'),
(123, 'sensor_2', 78.2, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00'),
(123, 'sensor_3', 120.8, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
"""
conn.execute(text(insert_sql))
print("[TEST] ✓ Data inserted successfully")
# Prepare input data for PredictionsBatch workflow
input_data = {
'metadata': {
'metadata': {
'model_id': 123,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}
},
'schedule_name': 'test-schedule',
'model_name': 'test_model',
'model_id': 123,
'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 123',
'schema': 'predictions_schema',
'table_name': 'predictions',
'transform_table_name': 'transformed_data',
'input_filters': {
'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
},
'mlflow_transform_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'mlflow_predict_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
'opc_output_config': {},
'pi_web_api_output_config': {},
'save_transform': True,
'prediction_store_policy': 'lts:1',
'model_config': {
'retention_minutes': 0,
'transform_flavor': 'sklearn',
'predict_flavor': 'sklearn',
},
'datetime_columns': ['timestamp', 'created_at'],
}
# Start workflow
print("\n[TEST] 2. Starting workflow...")
workflow_id = f'test-predictions-batch-{datetime.now().timestamp()}'
print(f"[TEST] Workflow ID: {workflow_id}")
handle = await client.start_workflow(
PredictionsBatch.run,
input_data,
id=workflow_id,
task_queue='test-queue',
)
print("[TEST] ✓ Workflow started")
# Wait for workflow completion with timeout
print("\n[TEST] 3. Waiting for workflow completion (timeout: 60s)...")
try:
await asyncio.wait_for(handle.result(), timeout=60.0) # 60 seconds timeout
print("[TEST] ✓ Workflow completed successfully")
except asyncio.TimeoutError:
print("[TEST] ✗ Workflow TIMEOUT after 60 seconds!")
pytest.fail("Workflow execution timed out after 60 seconds")
# Verify data was stored in PostgreSQL - use single connection
schema_name = 'predictions_schema'
predictions_table = 'predictions'
transformed_table = 'transformed_data'
full_predictions_table = f"{schema_name}.{predictions_table}"
full_transformed_table = f"{schema_name}.{transformed_table}"
# Use a single connection for all verification queries
print("\n[TEST] 4. Verifying results in PostgreSQL...")
with postgres_engine.connect() as conn:
# Verify prediction data
result_query = conn.execute(
text(f"SELECT model_id, prediction, prediction_confidence, response_time, prediction_status, comments FROM {full_predictions_table} WHERE model_id = 123")
)
prediction_rows = result_query.fetchall()
print(f"[TEST] Found {len(prediction_rows)} prediction record(s)")
assert len(prediction_rows) == 1, "Expected one prediction record"
# Verify first row has expected structure
row = prediction_rows[0]
print(f"[TEST] Prediction: {row}")
assert row[0] == 123, f"Expected model_id=123, got {row[0]}"
assert row[1] == 0.5, f"Expected prediction=0.5, got {row[1]}"
assert row[2] == 0, f"Expected prediction_confidence=0.9, got {row[2]}"
assert row[3] is not None, f"Expected response_time=0.1, got {row[3]}"
assert row[4] == 'Good', f"Expected prediction_status='Good', got {row[4]}"
assert row[5] == '', f"Expected comments='', got {row[5]}"
print("[TEST] ✓ Prediction data verified")
# Verify transformed data
result_query = conn.execute(
text(f"SELECT model_id, variable, value FROM {full_transformed_table} WHERE model_id = 123")
)
transformed_rows = result_query.fetchall()
print(f"[TEST] Found {len(transformed_rows)} transformed data record(s)")
assert len(transformed_rows) == 2, "Expected two transformed data records"
row_1 = transformed_rows[0]
print(f"[TEST] Transformed data: {row_1}")
assert row_1[0] == 123, f"Expected model_id=123, got {row_1[0]}"
assert row_1[1] == 'feature_1', f"Expected variable='sensor_1', got {row_1[1]}"
assert float(row_1[2]) == 0.234, f"Expected value=0.234, got {row_1[2]}"
row_2 = transformed_rows[1]
print(f"[TEST] Transformed data: {row_2}")
assert row_2[0] == 123, f"Expected model_id=123, got {row_2[0]}"
assert row_2[1] == 'feature_2', f"Expected variable='sensor_2', got {row_2[1]}"
assert float(row_2[2]) == 0.783, f"Expected value=0.783, got {row_2[2]}"
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_1_2_1_sql_query_execution_error(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 1.2.1: SQL Query Execution Error
Description:
SQL query fails due to syntax error or connection issue.
Expected Behavior:
- load_custom_query raises exception (caught by Temporal retry policy)
- Notification sent with SQL error details
- After retries, activity may return empty data or workflow may fail
- If empty data returned, workflow completes with early exit via input gate
Assertions:
- Error notification sent
- Workflow completes (either fails or exits early)
- No data in predictions table
"""
client = temporal_test_env.client
input_data = {
'metadata': {
'metadata': {
'model_id': 128,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}
},
'schedule_name': 'test-schedule',
'model_name': 'test_model',
'model_id': 128,
'query': 'SELECT * FROM nonexistent_table WHERE invalid_syntax =', # Invalid SQL
'schema': 'predictions_schema',
'table_name': 'predictions',
'transform_table_name': 'transformed_data',
'input_filters': {
'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
},
'mlflow_transform_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'mlflow_predict_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
'opc_output_config': {},
'pi_web_api_output_config': {},
'save_transform': True,
'prediction_store_policy': 'lts:1',
'model_config': {
'retention_minutes': 0,
'transform_flavor': 'sklearn',
'predict_flavor': 'sklearn',
},
}
print("\n[TEST] 1. Starting workflow with invalid SQL query...")
workflow_id = f'test-sql-error-{datetime.now().timestamp()}'
handle = await client.start_workflow(
PredictionsBatch.run,
input_data,
id=workflow_id,
task_queue='test-queue',
)
print("[TEST] ✓ Workflow started")
print("\n[TEST] 2. Waiting for workflow completion...")
try:
await asyncio.wait_for(handle.result(), timeout=60.0)
print("[TEST] ✓ Workflow completed (may have exited early due to empty data)")
except Exception as e:
print(f"[TEST] ✓ Workflow failed as expected: {type(e).__name__}")
# Verify no predictions were created (regardless of whether workflow failed or exited early)
print("\n[TEST] 3. Verifying no predictions were created...")
with postgres_engine.connect() as conn:
result_query = conn.execute(
text("SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = 128")
)
count = result_query.scalar()
assert count == 0, f"Expected no predictions, but found {count} records"
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_1_2_2_missing_required_parameters(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 1.2.2: Missing Required Parameters
Description:
Essential parameters missing from input.
Expected Behavior:
- Workflow or activity raises KeyError or validation error
- Workflow fails immediately
Assertions:
- Workflow fails with parameter error
- Error notification sent
- No child workflow called
"""
client = temporal_test_env.client
# Missing 'query' parameter
input_data = {
'metadata': {
'metadata': {
'model_id': 129,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}
},
'schedule_name': 'test-schedule',
'model_name': 'test_model',
'model_id': 129,
# 'query' is missing
'schema': 'predictions_schema',
'table_name': 'predictions',
'transform_table_name': 'transformed_data',
}
print("\n[TEST] 1. Starting workflow with missing required parameter...")
workflow_id = f'test-missing-param-{datetime.now().timestamp()}'
handle = await client.start_workflow(
PredictionsBatch.run,
input_data,
id=workflow_id,
task_queue='test-queue',
)
print("[TEST] ✓ Workflow started")
print("\n[TEST] 2. Waiting for workflow to fail...")
try:
await asyncio.wait_for(handle.result(), timeout=60.0)
pytest.fail("Expected workflow to fail, but it completed successfully")
except Exception as e:
print(f"[TEST] ✓ Workflow failed as expected: {type(e).__name__}")
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_1_2_3_invalid_datetime_column_specification(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 1.2.3: Invalid Datetime Column Specification
Description:
Datetime column specified doesn't exist in query results.
Expected Behavior:
- load_custom_query may raise KeyError or warning
- Depending on implementation, workflow may fail or continue
- Error notification sent
Assertions:
- Error raised or warning logged
- Workflow behavior depends on error handling policy
"""
client = temporal_test_env.client
print("\n[TEST] 1. Inserting test data...")
with postgres_engine.begin() as conn:
conn.execute(text("DELETE FROM predictions_schema.laborious_data WHERE model_id = 130"))
insert_sql = """
INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
VALUES
(130, 'sensor_1', 23.5, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
"""
conn.execute(text(insert_sql))
print("[TEST] ✓ Data inserted successfully")
input_data = {
'metadata': {
'metadata': {
'model_id': 130,
'model_name': 'test_model',
'schedule_name': 'test-schedule',
'workflow_name': 'predictions_batch',
}
},
'schedule_name': 'test-schedule',
'model_name': 'test_model',
'model_id': 130,
'query': 'SELECT timestamp, variable, value FROM predictions_schema.laborious_data WHERE model_id = 130',
'schema': 'predictions_schema',
'table_name': 'predictions',
'transform_table_name': 'transformed_data',
'input_filters': {
'EMPTY_DATA': {'policy': 'STOP', 'config': {}},
},
'mlflow_transform_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'mlflow_predict_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
'opc_output_config': {},
'pi_web_api_output_config': {},
'save_transform': True,
'prediction_store_policy': 'lts:1',
'model_config': {
'retention_minutes': 0,
'transform_flavor': 'sklearn',
'predict_flavor': 'sklearn',
},
'datetime_columns': ['nonexistent_column'], # Column doesn't exist in query result
}
print("\n[TEST] 2. Starting workflow with invalid datetime column...")
workflow_id = f'test-invalid-datetime-col-{datetime.now().timestamp()}'
handle = await client.start_workflow(
PredictionsBatch.run,
input_data,
id=workflow_id,
task_queue='test-queue',
)
print("[TEST] ✓ Workflow started")
print("\n[TEST] 3. Waiting for workflow completion or failure...")
try:
await asyncio.wait_for(handle.result(), timeout=60.0)
# Workflow may complete or fail depending on error handling
print("[TEST] ✓ Workflow completed (may have handled error gracefully)")
except Exception as e:
print(f"[TEST] ✓ Workflow failed as expected: {type(e).__name__}")
print("\n[TEST] ✓ Test completed!")

629
e2e/test_predictions_batch_prediction_process.py Normal file
View File

@@ -0,0 +1,629 @@
"""
End-to-end tests for PredictionsBatch workflow - Prediction Process scenarios.
"""
import asyncio
from datetime import datetime
from decimal import Decimal
from unittest.mock import MagicMock, patch
import pandas as pd
import pytest
from pytz import timezone
from sqlalchemy import text
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
from laborious.activities.activities import Activities
from laborious.workflows.predictions_batch import PredictionsBatch
base_input_data = {
'schedule_name': 'test-schedule',
'model_name': 'test_model',
'model_id': 201,
'query': 'SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = 201',
'schema': 'predictions_schema',
'table_name': 'predictions',
'transform_table_name': 'transformed_data',
'input_filters': {
'SPECIFIC_VARIABLES_NULL_VALUES': {
'policy': 'CONTINUE', # Continue despite issues, not STOP
'config': {'variables': ['sensor_1']},
},
},
'mlflow_transform_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'mlflow_predict_filters': {
'API_ERROR': {'policy': 'STOP', 'config': {}},
},
'path_priority': ['STOP', 'CONTINUE', 'REPEAT'],
'opc_output_config': {},
'pi_web_api_output_config': {},
'save_transform': True,
'prediction_store_policy': 'lts:1',
'model_config': {
'retention_minutes': 0,
'transform_flavor': 'sklearn',
'predict_flavor': 'sklearn',
},
'datetime_columns': ['timestamp', 'created_at'],
}
base_query = "SELECT timestamp, variable, value, created_at FROM predictions_schema.laborious_data WHERE model_id = {model_id}"
def get_base_input_data(model_id):
return {
**base_input_data,
'model_id': model_id,
'query': base_query.format(model_id=model_id),
}
def insert_sample_data(postgres_engine, model_id, values: list[tuple]):
with postgres_engine.begin() as conn:
conn.execute(text(f"DELETE FROM predictions_schema.laborious_data WHERE model_id = {model_id}"))
# Insert data with some null values (quality issue)
values_sql = []
for i, value in enumerate(values):
values_sql.append(f"""
({model_id}, 'sensor_{i+1}', {value}, '2024-01-01 12:00:00+00:00', '2024-01-01 12:00:00+00:00')
""")
insert_sql = f"""
INSERT INTO predictions_schema.laborious_data (model_id, variable, value, timestamp, created_at)
VALUES
{', '.join(values_sql)}
"""
conn.execute(text(insert_sql))
def insert_sample_prediction(postgres_engine, model_id):
with postgres_engine.begin() as conn:
conn.execute(text(f"DELETE FROM predictions_schema.predictions WHERE model_id = {model_id}"))
# Insert data with some null values (quality issue)
insert_sql = f"""
INSERT INTO predictions_schema.predictions (model_id, timestamp, prediction, prediction_confidence, prediction_status, comments, response_time)
VALUES
({model_id}, '2024-01-01 12:00:00+00:00', 10, 0, 'Good', '', 0.1)
"""
conn.execute(text(insert_sql))
return (model_id, Decimal(10), Decimal(0), 'Good')
async def start_and_await_workflow(client, input_data, workflow_id):
handle = await client.start_workflow(
PredictionsBatch.run,
input_data,
id=workflow_id,
task_queue='test-queue',
)
print("[TEST] ✓ Workflow started")
print("\n[TEST] 3. Waiting for workflow completion...")
try:
await asyncio.wait_for(handle.result(), timeout=60.0)
print("[TEST] ✓ Workflow completed successfully")
except asyncio.TimeoutError:
pytest.fail("Workflow execution timed out after 60 seconds")
def assert_continue(
postgres_engine, model_id, prediction_confidence: Decimal = 2,
comments: str = 'Input data with bad quality',
):
print("\n[TEST] 4. Verifying prediction was created despite warnings...")
with postgres_engine.connect() as conn:
result_query = conn.execute(
text(f"SELECT model_id, prediction, prediction_confidence, prediction_status, comments FROM predictions_schema.predictions WHERE model_id = {model_id}")
)
prediction_rows = result_query.fetchall()
assert len(prediction_rows) == 1, "Expected one prediction record despite warnings"
# Assert prediction value is 0 and other fields
row = prediction_rows[0]
assert row[1] == 0, f"Expected prediction=0, got {row[1]}"
assert row[2] == prediction_confidence, f"Expected prediction_confidence={prediction_confidence}, got {row[2]}"
assert row[3] == 'Bad', f"Expected prediction_status='Bad', got {row[3]}"
assert row[4] == comments, f"Expected comments='{comments}', got {row[4]}"
def assert_stop(postgres_engine, model_id):
print("\n[TEST] 4. Verifying no predictions were created...")
with postgres_engine.connect() as conn:
result_query = conn.execute(
text(f"SELECT COUNT(*) FROM predictions_schema.predictions WHERE model_id = {model_id}")
)
count = result_query.scalar()
assert count == 0, f"Expected no predictions, but found {count} records"
def assert_repeat(postgres_engine, model_id, last_prediction: list):
print("\n[TEST] 4. Verifying prediction was repeated...")
with postgres_engine.connect() as conn:
result_query = conn.execute(
text(f'SELECT model_id, prediction, prediction_confidence, prediction_status FROM predictions_schema.predictions WHERE model_id = {model_id}')
)
prediction_rows = result_query.fetchall()
print(prediction_rows)
assert len(prediction_rows) == 2, "Expected two prediction records"
assert prediction_rows[0] == last_prediction, f"Expected first prediction to be the same as the last prediction, got {prediction_rows[0]}, expected {last_prediction}"
assert prediction_rows[1] == last_prediction, f"Expected second prediction to be the same as the last prediction, got {prediction_rows[1]}, expected {last_prediction}"
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_1_1_input_gate_triggers_continue(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 2.1.1: Input Gate Triggers CONTINUE
Description:
Input gate determines data should use previous prediction.
Expected Behavior:
- input_gate returns path_flag='CONTINUE'
- path_flag_handler calls export workflow with input data directly
- MLFlow transform and predict skipped
- Data exported as-is
Assertions:
- input_gate called
- MLFlow operations NOT called
- Export workflow called with original data
- Workflow completes
"""
client = temporal_test_env.client
model_id = 211
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
print("\n[TEST] 2. Starting workflow with CONTINUE policy...")
workflow_id = f'test-continue-policy-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_continue(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_1_2_input_gate_triggers_stop(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 2.1.2: Input Gate Triggers STOP
Description:
Input data quality gate fails with STOP policy.
Expected Behavior:
- input_gate returns path_flag='STOP'
- path_flag_handler detects STOP
- Workflow returns early without calling MLFlow
- No prediction exported
Assertions:
- input_gate called
- path_flag_handler returns True (early exit)
- MLFlow transform NOT called
- Export workflow NOT called
- Workflow completes without error
"""
client = temporal_test_env.client
model_id = 212
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['policy'] = 'STOP'
print("\n[TEST] 2. Starting workflow that should stop at input gate...")
workflow_id = f'test-input-stop-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_stop(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_1_3_input_gate_triggers_repeat(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
):
"""
Scenario 2.1.3: Input Gate Triggers REPEAT
Description:
Input gate determines data should repeat last prediction.
Expected Behavior:
- input_gate returns path_flag='REPEAT'
- path_flag_handler calls repeat_last_prediction activity
- MLFlow transform and predict skipped
- Last prediction repeated and exported
Assertions:
- input_gate called
- MLFlow operations NOT called
- repeat_last_prediction activity called
- Workflow completes
"""
client = temporal_test_env.client
model_id = 213
print("\n[TEST] 1. Inserting test data and previous prediction...")
insert_sample_data(postgres_engine, model_id, ['NULL', 78.2])
data = insert_sample_prediction(postgres_engine, model_id)
print("[TEST] ✓ Data and previous prediction inserted")
input_data = get_base_input_data(model_id)
input_data['input_filters']['SPECIFIC_VARIABLES_NULL_VALUES']['policy'] = 'REPEAT'
print("\n[TEST] 2. Starting workflow that should trigger REPEAT...")
workflow_id = f'test-input-repeat-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_repeat(postgres_engine, model_id, data)
print("\n[TEST] ✓ All assertions passed!")
@pytest.fixture
def bad_data_model(patch_mlflow):
model = MagicMock(
predict=MagicMock(
side_effect=Exception("Bad data model")
)
)
patch_mlflow.sklearn.load_model = MagicMock(return_value=model)
return model
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_2_1_transform_gate_triggers_continue(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
bad_data_model,
):
"""
Scenario 2.2.1: Transform Gate Triggers CONTINUE
Description:
Transform response gate determines data should continue despite issues.
Expected Behavior:
- request_transform succeeds
- mlflow_response_gate for transform returns path_flag='CONTINUE'
- path_flag_handler calls export workflow with transform data
- MLFlow predict skipped
- Transform data exported as-is
Assertions:
- Transform completed
- mlflow_response_gate called for transform
- MLFlow predict NOT called
- Export workflow called with transform data
- Workflow completes
"""
client = temporal_test_env.client
model_id = 221
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
input_data = get_base_input_data(model_id)
input_data['mlflow_transform_filters']['API_ERROR']['policy'] = 'CONTINUE'
print("\n[TEST] 2. Starting workflow that should trigger CONTINUE at transform gate...")
workflow_id = f'test-transform-continue-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_continue(
postgres_engine=postgres_engine,
model_id=model_id,
prediction_confidence=Decimal(10),
comments='Bad data model',
)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_2_2_transform_gate_triggers_stop(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
bad_data_model,
):
"""
Scenario 2.2.2: Transform Gate Triggers STOP
Description:
Transform response validation fails with STOP policy.
Expected Behavior:
- request_transform succeeds but response invalid
- mlflow_response_gate for transform returns path_flag='STOP'
- Workflow exits without calling predict or export
Assertions:
- Transform completed but validation failed
- mlflow_response_gate called for transform
- MLFlow predict NOT called
- Export workflow NOT called
- Workflow completes without error
"""
client = temporal_test_env.client
model_id = 222
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['mlflow_transform_filters']['API_ERROR']['policy'] = 'STOP'
print("\n[TEST] 2. Starting workflow that should trigger STOP at transform gate...")
workflow_id = f'test-transform-stop-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_stop(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_2_3_transform_gate_triggers_repeat(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
bad_data_model,
):
"""
Scenario 2.2.3: Transform Gate Triggers REPEAT
Description:
Transform response gate determines data should repeat last prediction.
Expected Behavior:
- request_transform succeeds but response has issues
- mlflow_response_gate for transform returns path_flag='REPEAT'
- path_flag_handler calls repeat_last_prediction activity
- MLFlow predict skipped
- Last prediction repeated and exported
Assertions:
- Transform completed but validation triggered REPEAT
- mlflow_response_gate called for transform
- MLFlow predict NOT called
- repeat_last_prediction activity called
- Workflow completes
"""
client = temporal_test_env.client
model_id = 223
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
data = insert_sample_prediction(postgres_engine, model_id)
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['mlflow_transform_filters']['API_ERROR']['policy'] = 'REPEAT'
print("\n[TEST] 2. Starting workflow that should trigger REPEAT at transform gate...")
workflow_id = f'test-transform-repeat-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_repeat(postgres_engine, model_id, data)
print("\n[TEST] ✓ All assertions passed!")
@pytest.fixture
def bad_predict_model(
patch_mlflow,
mock_mlflow_models
):
model = MagicMock(
predict=MagicMock(
side_effect=Exception("Bad predict model")
)
)
def mock_sklearn_load_model(model_uri):
if 'data_model' in model_uri or 'transform' in model_uri.lower():
return mock_mlflow_models['transform_model']
return model
patch_mlflow.sklearn = MagicMock()
patch_mlflow.sklearn.load_model = MagicMock(side_effect=mock_sklearn_load_model)
return model
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_3_1_predict_gate_triggers_continue(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
bad_predict_model,
):
"""
Scenario 2.3.1: Predict Gate Triggers CONTINUE
Description:
Predict response gate determines data should continue despite issues.
Expected Behavior:
- request_predict succeeds
- mlflow_response_gate for predict returns path_flag='CONTINUE'
- path_flag_handler calls export workflow with predict data
- Prediction exported despite quality issues
Assertions:
- Transform and predict completed
- mlflow_response_gate called for predict
- Export workflow called with predict data
- Workflow completes
"""
client = temporal_test_env.client
model_id = 231
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [60.0, 78.2])
input_data = get_base_input_data(model_id)
input_data['mlflow_predict_filters']['API_ERROR']['policy'] = 'CONTINUE'
print("\n[TEST] 2. Starting workflow that should trigger CONTINUE at predict gate...")
workflow_id = f'test-predict-continue-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_continue(
postgres_engine=postgres_engine,
model_id=model_id,
prediction_confidence=Decimal(10),
comments='Bad predict model',
)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_3_2_predict_gate_triggers_stop(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
bad_predict_model,
):
"""
Scenario 2.3.2: Predict Gate Triggers STOP
Description:
Prediction validation fails with STOP policy.
Expected Behavior:
- request_predict succeeds but response invalid
- mlflow_response_gate for predict returns path_flag='STOP'
- Workflow exits without export
Assertions:
- Transform completed
- Predict completed but validation failed
- Export workflow NOT called
- Workflow completes without error
"""
client = temporal_test_env.client
model_id = 232
print("\n[TEST] 1. Inserting test data...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
print("[TEST] ✓ Data inserted successfully")
input_data = get_base_input_data(model_id)
input_data['mlflow_predict_filters']['API_ERROR']['policy'] = 'STOP'
print("\n[TEST] 2. Starting workflow that should stop at predict gate...")
workflow_id = f'test-predict-stop-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_stop(postgres_engine, model_id)
print("\n[TEST] ✓ All assertions passed!")
@pytest.mark.asyncio
@pytest.mark.integration
async def test_scenario_2_3_3_predict_gate_triggers_repeat(
temporal_test_env: WorkflowEnvironment,
temporal_worker: Worker,
test_activities: Activities,
postgres_engine,
bad_predict_model,
):
"""
Scenario 2.3.3: Predict Gate Triggers REPEAT
Description:
Predict response gate determines data should repeat last prediction.
Expected Behavior:
- request_predict succeeds but response has issues
- mlflow_response_gate for predict returns path_flag='REPEAT'
- path_flag_handler calls repeat_last_prediction activity
- Last prediction repeated and exported
Assertions:
- Transform and predict completed but validation triggered REPEAT
- mlflow_response_gate called for predict
- repeat_last_prediction activity called
- Export workflow NOT called with current prediction
- Workflow completes
"""
client = temporal_test_env.client
model_id = 233
print("\n[TEST] 1. Inserting test data and previous prediction...")
insert_sample_data(postgres_engine, model_id, [23.5, 78.2])
data = insert_sample_prediction(postgres_engine, model_id)
print("[TEST] ✓ Data and previous prediction inserted")
input_data = get_base_input_data(model_id)
input_data['mlflow_predict_filters']['API_ERROR']['policy'] = 'REPEAT'
input_data['path_priority'] = ['REPEAT', 'STOP', 'CONTINUE'] # REPEAT first
print("\n[TEST] 2. Starting workflow that should trigger REPEAT at predict gate...")
workflow_id = f'test-predict-repeat-{datetime.now().timestamp()}'
await start_and_await_workflow(client, input_data, workflow_id)
assert_repeat(postgres_engine, model_id, data)
print("\n[TEST] ✓ All assertions passed!")

2
git-requirements-mapping.txt Normal file
View File

@@ -0,0 +1,2 @@
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git:sientia-do
git+ssh://git@github.com/Aignosi/sientia-mlops-library.git:sientia

24
laborious/activities/activities.py
View File

@@ -7,6 +7,7 @@ with workflow.unsafe.imports_passed_through():
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.observability.logger import Logger
from laborious.activities.api import API
from laborious.activities.gates import Gates
from laborious.activities.mlflow import MLFlow
from laborious.activities.model_metrics import ModelMetrics
@@ -14,7 +15,7 @@ with workflow.unsafe.imports_passed_through():
from laborious.activities.storage import Storage
class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics, API):
"""
Main activities orchestrator for the Laborious system.
@@ -23,15 +24,18 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
MLFlow model interactions, data quality validation, and OPC server communications.
The class implements multiple inheritance to combine specialized functionality:
- Postgres: Database operations and data persistence
- Storage: Database operations and data persistence
- MLFlow: Model inference and transformation operations
- Gates: Data quality validation and filtering mechanisms
- OPC: Real-time data export to OPC servers
- ModelMetrics: Model performance metrics and drift detection
- API: PI Web API export operations for industrial systems
Attributes:
postgres_config (dict): PostgreSQL connection configuration
mlflow_config (dict): MLFlow server configuration
opc_config (dict): OPC server configuration
pi_web_api_config (dict): PI Web API server configuration
logger (Logger): Logging and observability instance
notification_handler (NotificationHandler): Notification management instance
"""
@@ -42,6 +46,7 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
mlflow_config: dict[str, Any],
minio_config: dict[str, Any],
opc_config: dict[str, Any],
pi_web_api_config: dict[str, Any],
logger: Logger,
notification_handler: NotificationHandler,
):
@@ -58,6 +63,8 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
Required keys: host, port, username, password
opc_config: OPC server configuration dictionary
Can contain multiple server configurations
pi_web_api_config: PI Web API server configuration dictionary
Required keys: base_url, auth_type, auth_token
logger: Logger instance for observability and debugging
notification_handler: Notification handler for alerts and monitoring
@@ -116,6 +123,16 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
metrics_controller=metrics_controller,
)
API.__init__(
self,
base_url=pi_web_api_config['base_url'],
auth_type=pi_web_api_config['auth_type'],
auth_token=pi_web_api_config['auth_token'],
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
async def shutdown(self):
"""
Gracefully shutdown all activities and clean up resources.
@@ -123,6 +140,8 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
This method ensures proper cleanup of all resources including:
- PostgreSQL connection pools
- OPC server connections
- PI Web API client connections
- MLFlow model repositories
- Any other resources that need explicit cleanup
The method should be called before the application terminates to ensure
@@ -133,3 +152,4 @@ class Activities(Storage, MLFlow, Gates, OPC, ModelMetrics):
Gates.close(self)
await OPC.close(self)
ModelMetrics.close(self)
API.close(self)

270
laborious/activities/api.py Normal file
View File

@@ -0,0 +1,270 @@
from temporalio import activity, workflow
with workflow.unsafe.imports_passed_through():
import json
import traceback
from typing import Any
from pandas import DataFrame
from sientia_do.notifications.handlers import NotificationHandler
from sientia_do.notifications.models import NotificationLevel
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
from sientia_do.repository.pi_web_api_client import PIWebAPIClient
from laborious import metrics
PI_WEB_API_PREDICTION_ERROR_CONFIDENCE = 13
class API(SientiaMonitoring):
"""
PI Web API operations for writing prediction data to PI Web API.
This class provides Temporal activities for interacting with the PI Web API
to write prediction and confidence values to industrial systems. It handles
error scenarios gracefully by setting error confidence values and sending
notifications when write operations fail.
The class implements comprehensive error handling for both prediction and
confidence value writes, ensuring that partial failures are properly
reported and handled.
"""
def __init__(
self,
base_url: str,
auth_type: str,
auth_token: str,
logger: Logger,
notification_handler: NotificationHandler,
metrics_controller: MetricsController,
) -> None:
"""
Initialize API activity with PI Web API client.
Args:
base_url (str): Base URL of the PI Web API server
auth_type (str): Authentication type ('basic' or 'bearer')
auth_token (str): Authentication token
logger (Logger): Logger instance for operation logging
notification_handler (NotificationHandler): Handler for system notifications
metrics_controller (MetricsController): Controller for metrics collection
"""
SientiaMonitoring.__init__(self, logger, notification_handler, metrics_controller)
self.pi_web_api_client = PIWebAPIClient(
base_url=base_url,
auth_config={
'type': auth_type,
'token': auth_token,
},
logger=logger,
notification_handler=notification_handler,
metrics_controller=metrics_controller,
)
def close(self) -> None:
"""
Close the PI Web API client and shutdown monitoring services.
This method properly closes all connections and resources associated
with the PI Web API client and monitoring services.
"""
self.pi_web_api_client.close()
SientiaMonitoring.shutdown(self)
async def process_pi_web_api_response(
self,
response_data: list[dict[str, Any]],
tags: dict[str, str],
core_labels: dict[str, str],
metadata: dict[str, Any],
) -> tuple[int, str]:
"""
Process the response data from PI Web API write operation.
Validates that all tags were successfully written, emits metrics for each tag
(success or error), and returns the appropriate prediction confidence value.
Sets error confidence if any tag write fails or if the number of written tags
doesn't match the expected count.
Args:
- response_data (dict[str, Any]): The response data from the PI Web API write operation.
- tags (dict[str, str]): The tags that were written to the PI Web API.
- core_labels (dict[str, str]): The core labels of the workflow execution.
- metadata (dict[str, Any]): The metadata of the workflow execution.
Returns:
int: Prediction confidence value (0 for success, 13 for errors)
"""
# Convert tags from name:webid to webid:name
tags = {w: t for t, w in tags.items()}
tag_names = list[str](tags.values())
confidence = 0
message = ''
# Evaluate response for each tag
written_tags = []
for item in response_data:
web_id = item.get('WebId')
if not web_id:
self.error('The response did not contain some WebIds', metadata)
continue
errors = item.get('Errors', [])
tag_name = tags.get(web_id)
if not tag_name:
self.error(
f'The response did not contain the tag name for WebId {web_id}', metadata
)
continue
if errors:
self.error(
f'Error writing tag {tag_name}:{web_id} to PI Web API: {errors}', metadata
)
await self.emit_metric(
metric_object=metrics.PI_WEB_API_PREDICTION_WRITTEN_ERROR_COUNT,
tags={
**core_labels,
'tag_name': tag_name,
},
)
confidence = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
else:
await self.emit_metric(
metric_object=metrics.PI_WEB_API_PREDICTION_WRITTEN_COUNT,
tags={
**core_labels,
'tag_name': tag_name,
},
)
written_tags.append(tag_name)
if len(written_tags) != len(tag_names):
message = f'The number of written tags does not match the number of tag names: Expected {tag_names} tags, but {written_tags} tags were written.'
self.error(
f'{message}\nResponse:\n {json.dumps(response_data, indent=4)}\nTags:\n {json.dumps(tags, indent=4)}',
metadata,
)
await self.send_notification_async(
metadata=metadata,
notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
message=f'The number of written tags does not match the number of tag names: Expected {tag_names} tags, but {written_tags} tags were written.\nResponse:\n {json.dumps(response_data, indent=4)}\nTags:\n {json.dumps(tags, indent=4)}',
block='write_pi_web_api_data',
level=NotificationLevel.ERROR,
)
confidence = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
return confidence, message
@activity.defn(name='write_pi_web_api_data')
async def write_pi_web_api_data(self, input_data: dict[str, Any]) -> dict[Any, Any]:
"""
Write prediction and confidence data to PI Web API.
Writes prediction values and confidence scores to PI Web API using configured
web IDs. Processes responses to validate writes and emit metrics. Handles errors
gracefully by setting error confidence values when writes fail and sending
notifications for both prediction and confidence write errors.
Args:
input_data (dict[str, Any]): The input data containing:
- metadata (dict[str, Any]): Workflow execution metadata
- pi_web_api_output_config (dict[str, Any]): PI Web API configuration with:
- endpoint (str): PI Web API endpoint URL
- prediction_tags (dict[str, str]): Mapping of tag names to web IDs for predictions
- confidence_tags (dict[str, str]): Mapping of tag names to web IDs for confidence
- data (dict[str, Any]): Prediction data, its a dataframe converted to dict.
Returns:
dict[Any, Any]: Data dictionary with potentially modified confidence values
If prediction write fails, prediction_confidence is set to error value (13)
"""
metadata = input_data['metadata']
data = DataFrame(input_data['data'])
pi_web_api_output_config = input_data['pi_web_api_output_config']
self.info(f'Writing data to PI Web API... config: {pi_web_api_output_config}', metadata)
raw_prediction_tags = pi_web_api_output_config['prediction_tags']
raw_confidence_tags = pi_web_api_output_config['confidence_tags']
prediction_tags = list[str](raw_prediction_tags.values())
confidence_tags = list(raw_confidence_tags.values())
core_labels = self.get_core_labels(metadata)
prediction_value = data.head(1)['prediction'].values[0]
confidence_value = data.head(1)['prediction_confidence'].values[0]
try:
prediction_response = await self.pi_web_api_client.write_value(
web_ids=prediction_tags,
value={
'Timestamp': data.head(1)['timestamp'].values[0],
'Value': prediction_value,
},
metadata=metadata,
)
confidence, message = await self.process_pi_web_api_response(
response_data=prediction_response,
tags=raw_prediction_tags,
core_labels=core_labels,
metadata=metadata,
)
data['prediction_confidence'] = confidence
data['comments'] = message
except Exception as e:
trace = traceback.format_exc()
await self.send_notification_async(
metadata=metadata,
notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
message=f'Error writing prediction data to PI Web API: {e}\n Tags: {raw_prediction_tags}',
block='write_pi_web_api_data',
level=NotificationLevel.ERROR,
attachment_content=trace,
)
self.error(trace, metadata)
data['prediction_confidence'] = PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
data['comments'] = str(e)
return data.to_dict()
try:
confidence_response = await self.pi_web_api_client.write_value(
web_ids=confidence_tags,
value={
'Timestamp': data.head(1)['timestamp'].values[0],
'Value': confidence_value,
},
metadata=metadata,
)
await self.process_pi_web_api_response(
response_data=confidence_response,
tags=raw_confidence_tags,
core_labels=core_labels,
metadata=metadata,
)
except Exception as e:
trace = traceback.format_exc()
await self.send_notification_async(
metadata=metadata,
notification_id='WRITE_PI_WEB_API_CONFIDENCE_ERROR',
message=f'Error writing confidence data to PI Web API: {e}\n Tags: {raw_confidence_tags}',
block='write_pi_web_api_data',
level=NotificationLevel.ERROR,
attachment_content=trace,
)
return data.to_dict()

2
laborious/activities/gates.py
View File

@@ -6,13 +6,13 @@ with workflow.unsafe.imports_passed_through():
from typing import Any
from pandas import DataFrame
from sientia_do.formatters import create_sample_dict
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.notifications.models import NotificationLevel
from sientia_do.observability.logger import Logger
from sientia_do.observability.metrics_controller import MetricsController
from sientia_do.observability.sientia_monitoring import SientiaMonitoring
from sientia_do.temporal.constants import DATETIME_FORMAT_WITH_TZ, now
from sientia_do.utils.formatters import create_sample_dict
from laborious import metrics
from laborious.utils.filters.conditional_filters import (

10
laborious/activities/mlflow.py
View File

@@ -6,7 +6,6 @@ with workflow.unsafe.imports_passed_through():
import numpy as np
from pandas import DataFrame, to_datetime
from sientia_do.formatters import create_sample_dict
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.notifications.models import NotificationLevel
from sientia_do.observability.logger import Logger
@@ -18,6 +17,7 @@ with workflow.unsafe.imports_passed_through():
DATETIME_FORMAT_WITH_TZ,
now,
)
from sientia_do.utils.formatters import create_sample_dict
from laborious.utils.repository.minio_repository import MinioRepository
from laborious.utils.repository.model_repository import MLFlowRepository
@@ -153,11 +153,11 @@ class MLFlow(SientiaMonitoring):
# Pivot data for model input format
data = data.pivot(index='timestamp', columns='variable', values='value')
data.fillna(np.nan, inplace=True)
# data.reset_index(inplace=True)
data.columns.name = None
self.debug('Processed input data:', metadata)
self.debug(data.head(5).to_string(), metadata)
data.columns.name = None
data.index.name = None
self.debug(f'Processed input data: \n {data.to_csv()}', metadata)
# Request transformation from MLFlow model
response_data = await self.model_monitoring_repository.transform(

5
laborious/activities/opc.py
View File

@@ -352,10 +352,13 @@ class OPC(SientiaMonitoring):
This allows downstream systems to handle data quality appropriately.
"""
message = 'Some data could not be written to OPC servers'
if not success:
data['prediction_confidence'] = OPC_WRITTING_ERROR_CONFIDENCE
data['comments'] = message
self.debug(
f'Some data could not be written to OPC servers, setting confidence to {OPC_WRITTING_ERROR_CONFIDENCE}.',
f'{message}, setting confidence to {OPC_WRITTING_ERROR_CONFIDENCE}.',
metadata,
)

21
laborious/metrics.py
View File

@@ -24,7 +24,9 @@ Metric Labels:
"""
from prometheus_client import Counter, Gauge, Histogram
from sientia_do.observability.metrics import CORE_LABELS as SIENTIA_CORE_LABELS
from sientia_do.observability.metrics import (
CORE_LABELS as SIENTIA_CORE_LABELS,
)
# Application health metric
APP_UP = Gauge(
@@ -188,3 +190,20 @@ MODEL_ANALYZE_ERROR_COUNT = Counter(
'Number of errors during analyze operations',
SIENTIA_CORE_LABELS,
)
# ================== PI Web API metrics ==================
PI_WEB_API_LABELS = [*CORE_LABELS, 'tag_name']
PI_WEB_API_PREDICTION_WRITTEN_COUNT = Counter(
'laborious_pi_web_api_prediction_written_count',
'Number of predictions written to the PI Web API',
PI_WEB_API_LABELS,
)
PI_WEB_API_PREDICTION_WRITTEN_ERROR_COUNT = Counter(
'laborious_pi_web_api_prediction_written_error_count',
'Number of errors writing predictions to the PI Web API',
PI_WEB_API_LABELS,
)

62
laborious/utils/connectors_config.py
View File

@@ -3,37 +3,6 @@ from os import getenv
from typing import Any
def build_postgres_config() -> dict[str, Any]:
"""
Build PostgreSQL database configuration from environment variables.
This function constructs a PostgreSQL configuration dictionary from
environment variables with sensible defaults for local development.
It handles connection pool configuration and security parameters.
Environment Variables:
POSTGRES_HOST: Database hostname (default: localhost)
POSTGRES_PORT: Database port (default: 5432)
POSTGRES_USER: Database username (default: sientia)
POSTGRES_PASSWORD: Database password (default: sientia)
POSTGRES_DBNAME: Database name (default: sientia)
POSTGRES_MIN_CONNECTIONS: Minimum connection pool size (default: 5)
POSTGRES_MAX_CONNECTIONS: Maximum connection pool size (default: 20)
Returns:
dict: PostgreSQL configuration dictionary with all required parameters
"""
return {
'host': getenv('POSTGRES_HOST', 'localhost'),
'port': int(getenv('POSTGRES_PORT', '5432')),
'user': getenv('POSTGRES_USER', 'sientia'),
'password': getenv('POSTGRES_PASSWORD', 'sientia'),
'dbname': getenv('POSTGRES_DBNAME', 'sientia'),
'min_connections': int(getenv('POSTGRES_MIN_CONNECTIONS', '5')),
'max_connections': int(getenv('POSTGRES_MAX_CONNECTIONS', '20')),
}
def build_mlflow_config() -> dict[str, Any]:
"""
Build MLFlow server configuration from environment variables.
@@ -99,37 +68,6 @@ def build_opc_config() -> dict[str, Any]:
}
def build_mongodb_config() -> dict[str, Any]:
"""
Build MongoDB configuration from environment variables.
This function constructs a MongoDB configuration dictionary from
environment variables with sensible defaults for local development.
It handles connection string and database name configuration.
Environment Variables:
MONGODB_USERNAME: MongoDB username (default: root)
MONGODB_PASSWORD: MongoDB password (default: wKZDbMNU1c)
MONGODB_URL: MongoDB connection URI (default: localhost:27018)
MONGODB_DATABASE_NAME: MongoDB database name (default: sientia)
MONGODB_TTL_INDEX_HOURS: TTL index duration in hours (default: 1)
Returns:
dict: MongoDB configuration dictionary with connection parameters
"""
username = getenv('MONGODB_USERNAME', 'root')
password = getenv('MONGODB_PASSWORD', 'wKZDbMNU1c')
uri = getenv('MONGODB_URL', 'localhost:27018')
connection_string = f'mongodb://{username}:{password}@{uri}'
return {
'connection_string': connection_string,
'database_name': getenv('MONGODB_DATABASE_NAME', 'sientia'),
'ttl_index_seconds': int(getenv('MONGODB_TTL_INDEX_HOURS', '1')) * 3600,
}
def build_minio_config() -> dict[str, Any]:
"""
Build MinIO (S3-compatible) configuration from environment variables.

17
laborious/utils/repository/model_repository.py
View File

@@ -752,6 +752,7 @@ class MLFlowRepository(SientiaMonitoring):
latest_production_id: str,
metadata: dict,
transform_flavor: str = 'sklearn',
skip_transform: bool = False,
predict_flavor: str = 'sklearn',
target_name: str | None = None,
) -> dict[str, Any]:
@@ -812,7 +813,10 @@ class MLFlowRepository(SientiaMonitoring):
load_wrapper=load_predict_wrapper,
)
treated_data_candidate = data_model.fit(data)
if not skip_transform:
treated_data_candidate = data_model.fit(data)
else:
treated_data_candidate = data_model
if not isinstance(treated_data_candidate, pd.DataFrame):
data_model = treated_data_candidate
@@ -1164,7 +1168,7 @@ class MLFlowRepository(SientiaMonitoring):
and returned in the response structure rather than propagated.
"""
self.debug(f'Data received for model transformation: {data.head(5).to_csv()}', metadata)
self.debug(f'Data received for model transformation: {data.to_csv()}', metadata)
# data.to_csv(
# f"tmp/data_{model_name}.csv", index=True)
@@ -1250,7 +1254,9 @@ class MLFlowRepository(SientiaMonitoring):
input_index = data.index
start_time = datetime.now()
self.debug(f'Data received for model prediction: {data.head(5).to_csv()}', metadata)
self.debug(
f'Data received for model prediction: {data.to_dict(orient="records")}', metadata
)
# data.to_csv(
# f"tmp/treated_data_{model_name}.csv", index=True)
@@ -1268,7 +1274,8 @@ class MLFlowRepository(SientiaMonitoring):
if isinstance(predict_data, pd.DataFrame):
self.debug(
f'Data received from model prediction: {data.head(5).to_csv()}', metadata
f'Data received from model prediction: {predict_data.to_dict(orient="records")}',
metadata,
)
# predict_data.to_csv(
@@ -1343,6 +1350,7 @@ class MLFlowRepository(SientiaMonitoring):
transform_flavor = model_config.get('transform_flavor', 'sklearn')
predict_flavor = model_config.get('predict_flavor', 'sklearn')
skip_transform = model_config.get('skip_transform', False)
self.debug(
f'Model configuration - transform_flavor: {transform_flavor}, predict_flavor: {predict_flavor}, target_name: {target_name}',
@@ -1356,6 +1364,7 @@ class MLFlowRepository(SientiaMonitoring):
model_name=model_name,
data=data,
transform_flavor=transform_flavor,
skip_transform=skip_transform,
predict_flavor=predict_flavor,
target_name=target_name,
metadata=metadata,

72
laborious/worker/prepare_worker.py Normal file
View File

@@ -0,0 +1,72 @@
import os
import re
from collections.abc import Sequence
from typing import Any
from sientia_do.observability.logger import Logger
from temporalio.client import Client
from temporalio.worker import PollerBehaviorAutoscaling, Worker
parameters = [
('MAX_CONCURRENT_WORKFLOW_TASKS', '200'),
('MAX_CONCURRENT_ACTIVITIES', '200'),
('MAX_CONCURRENT_LOCAL_ACTIVITIES', '200'),
('MAX_CACHED_WORKFLOWS', '200'),
('WORKFLOW_POLLER_BEHAVIOUR_MINIMUM', '10'),
('WORKFLOW_POLLER_BEHAVIOUR_INITIAL', '100'),
('WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM', '200'),
('ACTIVITY_POLLER_BEHAVIOUR_MINIMUM', '10'),
('ACTIVITY_POLLER_BEHAVIOUR_INITIAL', '100'),
('ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM', '200'),
]
def camel_to_snake(text: str) -> str:
"""Convert camelCase or PascalCase to snake_case."""
text = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', text)
text = re.sub('([a-z0-9])([A-Z])', r'\1_\2', text)
return text.lower()
def prepare_worker(
main_workflow: type,
other_workflows: Sequence[type],
activities: Sequence[Any],
temporal_client: Client,
logger: Logger,
) -> Worker:
main_workflow_name = main_workflow.__name__.upper()
queue_name = f'{camel_to_snake(main_workflow.__name__)}-queue'
local_workflow_parameters = {}
for parameter in parameters:
local_workflow_parameters[parameter[0]] = int(
os.getenv(main_workflow_name + '_' + parameter[0], parameter[1])
)
logger.info(f'Preparing worker for {main_workflow_name} with queue {queue_name}')
return Worker(
temporal_client,
task_queue=queue_name,
workflows=[main_workflow, *other_workflows],
activities=[*activities],
max_concurrent_workflow_tasks=local_workflow_parameters['MAX_CONCURRENT_WORKFLOW_TASKS'],
max_concurrent_activities=local_workflow_parameters['MAX_CONCURRENT_ACTIVITIES'],
max_concurrent_local_activities=local_workflow_parameters[
'MAX_CONCURRENT_LOCAL_ACTIVITIES'
],
max_cached_workflows=local_workflow_parameters['MAX_CACHED_WORKFLOWS'],
workflow_task_poller_behavior=PollerBehaviorAutoscaling(
minimum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MINIMUM'],
initial=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_INITIAL'],
maximum=local_workflow_parameters['WORKFLOW_POLLER_BEHAVIOUR_MAXIMUM'],
),
activity_task_poller_behavior=PollerBehaviorAutoscaling(
minimum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MINIMUM'],
initial=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_INITIAL'],
maximum=local_workflow_parameters['ACTIVITY_POLLER_BEHAVIOUR_MAXIMUM'],
),
)

147
laborious/worker/worker.py
View File

@@ -7,6 +7,7 @@ prediction and retraining workflows.
The worker supports multiple task queues:
- predictions_batch-queue: Handles batch prediction workflows (heavy workload)
Includes activities for MLFlow, data quality gates, OPC export, PI Web API export, and PostgreSQL
- minimal_retrain-queue: Handles model retraining workflows
- drift-queue: Handles drift detection workflows
- simple_metrics-queue: Handles simple metrics calculation workflows
@@ -26,51 +27,33 @@ Environment Variables:
- HTTP_METRICS_PORT: Prometheus metrics server port (default: 9090)
- HTTP_SDK_METRICS_PORT: Temporal SDK metrics port (default: 9091)
- PROJECT_NAME: Project name for notifications (default: laborious)
Tuner Configuration (Resource-based scaling):
- TUNER_TARGET_MEMORY_USAGE: Target memory usage (0.0-1.0, default: 0.75)
- TUNER_TARGET_CPU_USAGE: Target CPU usage (0.0-1.0, default: 0.80)
- TUNER_WORKFLOW_MIN_SLOTS: Minimum workflow slots (default: 5)
- TUNER_WORKFLOW_MAX_SLOTS: Maximum workflow slots (default: 50)
- TUNER_ACTIVITY_MIN_SLOTS: Minimum activity slots (default: 5)
- TUNER_ACTIVITY_MAX_SLOTS: Maximum activity slots (default: 50)
- TUNER_WORKFLOW_RAMP_THROTTLE_MS: Workflow ramp throttle in ms (default: 100)
- TUNER_ACTIVITY_RAMP_THROTTLE_MS: Activity ramp throttle in ms (default: 50)
Poller Configuration:
- POLLER_MINIMUM: Minimum number of pollers (default: 1)
- POLLER_MAXIMUM: Maximum number of pollers (default: 10)
- POLLER_INITIAL: Initial number of pollers (default: 2)
"""
from temporalio import client, workflow
from temporalio.runtime import PrometheusConfig, Runtime, TelemetryConfig
from temporalio.worker import (
PollerBehaviorAutoscaling,
ResourceBasedSlotConfig,
Worker,
WorkerTuner,
)
with workflow.unsafe.imports_passed_through():
import asyncio
import os
import sys
from datetime import timedelta
from prometheus_client import start_http_server
from sientia_do.notifications.handlers import CoreNotificationHandler as NotificationHandler
from sientia_do.observability.logger import get_logger
from sientia_do.utils.connectors_config import (
build_api_config,
build_mongodb_config,
build_postgres_config,
)
from laborious import metrics
from laborious.activities.activities import Activities
from laborious.utils.connectors_config import (
build_minio_config,
build_mlflow_config,
build_mongodb_config,
build_opc_config,
build_postgres_config,
)
from laborious.worker.prepare_worker import prepare_worker
from laborious.workflows.drift import Drift
from laborious.workflows.minimal_retrain import MinimalRetrain
from laborious.workflows.predictions_batch import PredictionsBatch
@@ -79,55 +62,11 @@ with workflow.unsafe.imports_passed_through():
FormatAndExportPrediction,
)
from laborious.workflows.sub_workflows.prediction_process import PredictionProcess
import os
POD_ID = os.getenv('POD_ID')
SDK_METRICS_PORT = int(os.getenv('HTTP_SDK_METRICS_PORT', '9091'))
def create_resource_tuner() -> WorkerTuner:
"""Create a resource-based tuner from environment variables."""
target_memory = float(os.getenv('TUNER_TARGET_MEMORY_USAGE', '0.75'))
target_cpu = float(os.getenv('TUNER_TARGET_CPU_USAGE', '0.50'))
workflow_min = int(os.getenv('TUNER_WORKFLOW_MIN_SLOTS', '5'))
workflow_max = int(os.getenv('TUNER_WORKFLOW_MAX_SLOTS', '50'))
activity_min = int(os.getenv('TUNER_ACTIVITY_MIN_SLOTS', '5'))
activity_max = int(os.getenv('TUNER_ACTIVITY_MAX_SLOTS', '50'))
local_activity_min = int(os.getenv('TUNER_LOCAL_ACTIVITY_MIN_SLOTS', '1'))
local_activity_max = int(os.getenv('TUNER_LOCAL_ACTIVITY_MAX_SLOTS', '30'))
workflow_ramp = int(os.getenv('TUNER_WORKFLOW_RAMP_THROTTLE_MS', '100'))
activity_ramp = int(os.getenv('TUNER_ACTIVITY_RAMP_THROTTLE_MS', '50'))
local_activity_ramp = int(os.getenv('TUNER_LOCAL_ACTIVITY_RAMP_THROTTLE_MS', '50'))
return WorkerTuner.create_resource_based(
target_memory_usage=target_memory,
target_cpu_usage=target_cpu,
workflow_config=ResourceBasedSlotConfig(
minimum_slots=workflow_min,
maximum_slots=workflow_max,
ramp_throttle=timedelta(milliseconds=workflow_ramp),
),
activity_config=ResourceBasedSlotConfig(
minimum_slots=activity_min,
maximum_slots=activity_max,
ramp_throttle=timedelta(milliseconds=activity_ramp),
),
local_activity_config=ResourceBasedSlotConfig(
minimum_slots=local_activity_min,
maximum_slots=local_activity_max,
ramp_throttle=timedelta(milliseconds=local_activity_ramp),
),
)
def create_poller_behavior() -> PollerBehaviorAutoscaling:
"""Create poller behavior from environment variables."""
minimum = int(os.getenv('POLLER_MINIMUM', '1'))
maximum = int(os.getenv('POLLER_MAXIMUM', '10'))
initial = int(os.getenv('POLLER_INITIAL', '2'))
return PollerBehaviorAutoscaling(minimum=minimum, maximum=maximum, initial=initial)
async def main():
"""
Main entry point for the Laborious worker application.
@@ -181,6 +120,7 @@ async def main():
mlflow_config=build_mlflow_config(),
minio_config=build_minio_config(),
opc_config=build_opc_config(),
pi_web_api_config=build_api_config(),
logger=logger,
notification_handler=notification_handler,
)
@@ -206,14 +146,11 @@ async def main():
logger.custom_info('Starting Workers...', metadata)
tuner = create_resource_tuner()
poller = create_poller_behavior()
workers = [
Worker(
temporal_client,
task_queue='minimal_retrain-queue',
workflows=[MinimalRetrain],
prepare_worker(
temporal_client=temporal_client,
main_workflow=MinimalRetrain,
other_workflows=[],
activities=[
activities.load_custom_query,
activities.query_to_minio,
@@ -222,44 +159,35 @@ async def main():
activities.format_retrain_report,
activities.export_data_to_postgres,
],
tuner=tuner,
max_cached_workflows=2,
workflow_task_poller_behavior=poller,
activity_task_poller_behavior=poller,
logger=logger,
),
Worker(
temporal_client,
task_queue='drift-queue',
workflows=[Drift],
prepare_worker(
temporal_client=temporal_client,
main_workflow=SimpleMetrics,
other_workflows=[],
activities=[
activities.load_custom_query,
activities.calculate_simple_metrics,
activities.export_data_to_postgres,
],
logger=logger,
),
prepare_worker(
temporal_client=temporal_client,
main_workflow=Drift,
other_workflows=[],
activities=[
activities.load_custom_query,
activities.get_reference_data,
activities.calculate_drift,
activities.export_data_to_postgres,
],
tuner=tuner,
max_cached_workflows=2,
workflow_task_poller_behavior=poller,
activity_task_poller_behavior=poller,
logger=logger,
),
Worker(
temporal_client,
task_queue='simple_metrics-queue',
workflows=[SimpleMetrics],
activities=[
activities.load_custom_query,
activities.calculate_simple_metrics,
activities.export_data_to_postgres,
],
tuner=tuner,
max_cached_workflows=2,
workflow_task_poller_behavior=poller,
activity_task_poller_behavior=poller,
),
Worker(
temporal_client,
task_queue='predictions_batch-queue',
workflows=[PredictionsBatch, PredictionProcess, FormatAndExportPrediction],
prepare_worker(
temporal_client=temporal_client,
main_workflow=PredictionsBatch,
other_workflows=[PredictionProcess, FormatAndExportPrediction],
activities=[
# MLFlow
activities.request_predict,
@@ -279,11 +207,10 @@ async def main():
activities.repeat_last_prediction,
activities.export_data_to_postgres,
activities.write_metrics,
# Pi Web API
activities.write_pi_web_api_data,
],
tuner=tuner,
max_cached_workflows=200,
workflow_task_poller_behavior=poller,
activity_task_poller_behavior=poller,
logger=logger,
),
]

5
laborious/workflows/predictions_batch.py
View File

@@ -61,7 +61,10 @@ class PredictionsBatch:
- model_retention (int, optional): Model retention period in minutes
- path_priority (list[str]): Decision path priority configuration
- opc_output_config (dict, optional): OPC server export configuration
- pi_web_api_output_config (dict, optional): PI Web API export configuration
- datetime_columns (list[str], optional): Columns to treat as datetime
- save_transform (bool, optional): Whether to save transformed data (default: True)
- prediction_store_policy (str, optional): Data retention policy (default: 'lts:1')
Returns:
None: The workflow completes successfully when the child workflow finishes
@@ -111,7 +114,9 @@ class PredictionsBatch:
'model_config': input_data.get('model_config', {}),
'path_priority': input_data.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT']),
'opc_output_config': input_data.get('opc_output_config', {}),
'pi_web_api_output_config': input_data.get('pi_web_api_output_config', {}),
'prediction_store_policy': input_data.get('prediction_store_policy', 'lts:1'),
'save_transform': input_data.get('save_transform', True),
}
# Execute prediction process workflow

55
laborious/workflows/sub_workflows/format_and_export_prediction.py
View File

@@ -26,6 +26,7 @@ class FormatAndExportPrediction:
Export Destinations:
- PostgreSQL Database: Persistent storage with timestamp conversion
- PI Web API: Real-time industrial system integration for prediction and confidence values
- OPC Servers: Real-time industrial system integration
- Prometheus Metrics: Performance monitoring and operational visibility
"""
@@ -38,9 +39,10 @@ class FormatAndExportPrediction:
This method orchestrates the complete data export process by:
1. Determining the appropriate formatting strategy based on path_flag
2. Formatting prediction data according to quality and requirements
3. Exporting data to OPC servers for real-time industrial access
4. Persisting data to PostgreSQL database with comprehensive metadata
5. Recording performance metrics for operational monitoring
3. Exporting data to PI Web API for real-time industrial access (if configured)
4. Exporting data to OPC servers for real-time industrial access (if configured)
5. Persisting data to PostgreSQL database with comprehensive metadata
6. Recording performance metrics for operational monitoring
The method implements flexible formatting strategies:
- Normal predictions: Full data formatting with confidence scores
@@ -61,8 +63,10 @@ class FormatAndExportPrediction:
- model_name (str): Name of the ML model
- schema (str): Database schema for data storage
- table_name (str): Target table for data persistence
- opc_output_config (dict[str, Any]): OPC server export configuration
Optional keys:
- opc_output_config (dict[str, Any]): OPC server export configuration
- pi_web_api_output_config (dict[str, Any]): PI Web API export configuration
Contains endpoint, prediction_tags, and confidence_tags mappings
- transformed_data (dict[str, Any]): Transformed data to export separately
Only processed when path_flag is None
- transform_table_name (str): Target table for transformed data export
@@ -87,6 +91,9 @@ class FormatAndExportPrediction:
transformed_data = input_data.get('transformed_data', None)
prediction_confidence = input_data['prediction_confidence']
opc_output_config = input_data.get('opc_output_config', None)
pi_web_api_output_config = input_data.get('pi_web_api_output_config', None)
if path_flag is None:
# Normal prediction path: format prediction data with full metadata
prediction = await workflow.execute_local_activity_method(
@@ -152,20 +159,36 @@ class FormatAndExportPrediction:
write_transformed_handler = None
opc_metrics = {}
# write to pi web api
if pi_web_api_output_config:
prediction = await workflow.execute_activity_method(
Activities.write_pi_web_api_data,
{
'pi_web_api_output_config': pi_web_api_output_config,
'data': prediction,
**metadata,
},
retry_policy=retry_policy,
start_to_close_timeout=timedelta(seconds=60),
)
# write to opc
prediction, opc_metrics = await workflow.execute_activity_method(
Activities.write_opc_data,
{
'opc_output_config': input_data['opc_output_config'],
'data': prediction,
**metadata,
},
retry_policy=retry_policy,
start_to_close_timeout=timedelta(seconds=60),
)
if opc_output_config:
prediction, opc_metrics = await workflow.execute_activity_method(
Activities.write_opc_data,
{
'opc_output_config': opc_output_config,
'data': prediction,
**metadata,
},
retry_policy=retry_policy,
start_to_close_timeout=timedelta(seconds=60),
)
# write to postgres
prediction_handler = workflow.execute_activity_method(
await workflow.execute_activity_method(
Activities.export_data_to_postgres,
{
**metadata,
@@ -178,8 +201,6 @@ class FormatAndExportPrediction:
start_to_close_timeout=timedelta(seconds=180),
)
await prediction_handler
if write_transformed_handler is not None:
await write_transformed_handler

21
laborious/workflows/sub_workflows/prediction_process.py
View File

@@ -66,7 +66,10 @@ class PredictionProcess:
- mlflow_predict_filters (dict): MLFlow prediction filters
- model_retention (int): Model retention period in minutes
- path_priority (list[str]): Decision path priority configuration
- opc_output_config (dict): OPC server export configuration
- opc_output_config (dict, optional): OPC server export configuration
- pi_web_api_output_config (dict, optional): PI Web API export configuration
- save_transform (bool, optional): Whether to save transformed data (default: True)
- prediction_store_policy (str, optional): Data retention policy (default: 'lts:1')
Returns:
None: The workflow completes successfully when export workflow finishes
@@ -207,6 +210,7 @@ class PredictionProcess:
'model_name': model_name,
'model_config': model_config,
'opc_output_config': input_data['opc_output_config'],
'pi_web_api_output_config': input_data['pi_web_api_output_config'],
'schema': input_data['schema'],
'table_name': input_data['table_name'],
'transform_table_name': input_data['transform_table_name'],
@@ -234,7 +238,17 @@ class PredictionProcess:
Args:
data: Input data for processing
path_flag: Path decision from filter (STOP, CONTINUE, REPEAT)
input_data: Complete workflow input configuration
input_data: Complete workflow input configuration including:
- metadata (dict): Workflow execution metadata
- schema (str): Database schema
- table_name (str): Target table for predictions
- transform_table_name (str): Target table for transformed data
- model_id (str): ML model identifier
- model_name (str): ML model name
- model_config (dict, optional): Model configuration
- opc_output_config (dict, optional): OPC server export configuration
- pi_web_api_output_config (dict, optional): PI Web API export configuration
- prediction_store_policy (str, optional): Data retention policy
confidence: Confidence level from filter validation
last_timestamp: Last processed timestamp
comment: Additional information about the filter result
@@ -244,7 +258,7 @@ class PredictionProcess:
Path Handling:
- STOP: Terminates workflow execution
- CONTINUE: Proceeds with normal processing
- CONTINUE: Delegates to FormatAndExportPrediction workflow with current data
- REPEAT: Repeats last prediction if available
"""
metadata = input_data['metadata']
@@ -294,6 +308,7 @@ class PredictionProcess:
'transform_table_name': transform_table_name,
'comment': comment,
'opc_output_config': input_data['opc_output_config'],
'pi_web_api_output_config': input_data['pi_web_api_output_config'],
'prediction_store_policy': input_data['prediction_store_policy'],
},
)

2
pyproject.toml
View File

@@ -123,7 +123,7 @@ markers = [
]
[tool.coverage.run]
source = ["model_manager"]
source = ["laborious"]
omit = [
"*/tests/*",
"*/venv/*",

3
requirements-dev.txt
View File

@@ -13,7 +13,8 @@ types-requests>=2.31.0 # Type stubs for requests
pytest>=7.4.0 # Testing framework
pytest-cov>=4.1.0 # Coverage plugin for pytest
pytest-asyncio>=0.21.0 # Async test support (already in main requirements)
testcontainers[postgres] # PostgreSQL containers for E2E tests
# Development Tools
ipython>=8.12.0 # Enhanced Python shell
ipdb>=0.13.13 # IPython debugger
ipdb>=0.13.13 # IPython debugger

2
requirements-light.txt
View File

@@ -3,7 +3,7 @@ psycopg2-binary
sqlalchemy
asyncua
redis
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.6.1
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.8.0
prometheus-client
botocore
boto3

7
requirements.txt
View File

@@ -3,8 +3,8 @@ psycopg2-binary
sqlalchemy
asyncua
redis
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.6.1
git+ssh://git@github.com/Aignosi/sientia-mlops-library.git@0.40.6
git+ssh://git@github.com/Aignosi/sientia-dataops-library.git@1.8.2
git+ssh://git@github.com/Aignosi/sientia-mlops-library.git@0.40.7
prometheus-client
botocore
boto3
@@ -12,4 +12,5 @@ s3fs
pyarrow
kaleido
hyperopt
shap
shap
pycurl

2
sonar-project.properties
View File

@@ -3,7 +3,7 @@ sonar.projectName=sientia-dataops-laborious_temporal
sonar.sources=laborious
sonar.tests=tests
sonar.projectVersion=1.0.0
sonar.coverage.exclusions=laborious/worker/worker.py
sonar.coverage.exclusions=laborious/worker/*
sonar.qualitygate.wait=true
sonar.qualitygate.timeout=300
sonar.python.coverage.reportPaths=coverage.xml

1045
tests.ipynb
View File

File diff suppressed because one or more lines are too long

58
tests/laborious/activities/test_activities.py
View File

@@ -3,8 +3,10 @@ from unittest.mock import ANY, AsyncMock, MagicMock, patch
from pytest import mark
from laborious.activities.activities import Activities
from laborious.activities.api import API
from laborious.activities.gates import Gates
from laborious.activities.mlflow import MLFlow
from laborious.activities.model_metrics import ModelMetrics
from laborious.activities.opc import OPC
from laborious.activities.storage import Storage
@@ -13,9 +15,17 @@ from laborious.activities.storage import Storage
@patch('laborious.activities.activities.MLFlow.__init__')
@patch('laborious.activities.activities.OPC.__init__')
@patch('laborious.activities.activities.Gates.__init__')
@patch('laborious.activities.activities.ModelMetrics.__init__')
@patch('laborious.activities.activities.API.__init__')
@patch('laborious.activities.activities.MetricsController')
def test___init__(
mock_metrics_controller, mock_gates_init, mock_opc_init, mock_mlflow_init, mock_storage_init
mock_metrics_controller,
mock_api_init,
mock_model_metrics_init,
mock_gates_init,
mock_opc_init,
mock_mlflow_init,
mock_storage_init,
):
postgres_config = {
'host': 'localhost',
@@ -43,6 +53,12 @@ def test___init__(
'group_id': 'test-group',
}
pi_web_api_config = {
'base_url': 'https://test-pi-server.com',
'auth_type': 'bearer',
'auth_token': 'test_token',
}
logger = MagicMock()
notification_handler = MagicMock()
@@ -51,6 +67,7 @@ def test___init__(
mlflow_config=mlflow_config,
minio_config=minio_config,
opc_config=opc_config,
pi_web_api_config=pi_web_api_config,
logger=logger,
notification_handler=notification_handler,
)
@@ -60,6 +77,8 @@ def test___init__(
assert isinstance(activities, MLFlow)
assert isinstance(activities, OPC)
assert isinstance(activities, Gates)
assert isinstance(activities, ModelMetrics)
assert isinstance(activities, API)
mock_storage_init.assert_called_once_with(
ANY,
@@ -103,13 +122,39 @@ def test___init__(
metrics_controller=mock_metrics_controller.return_value,
)
mock_model_metrics_init.assert_called_once_with(
ANY,
logger=logger,
notification_handler=notification_handler,
metrics_controller=mock_metrics_controller.return_value,
)
mock_api_init.assert_called_once_with(
ANY,
base_url=pi_web_api_config['base_url'],
auth_type=pi_web_api_config['auth_type'],
auth_token=pi_web_api_config['auth_token'],
logger=logger,
notification_handler=notification_handler,
metrics_controller=mock_metrics_controller.return_value,
)
@mark.asyncio
@patch('laborious.activities.activities.Storage')
@patch('laborious.activities.activities.MLFlow')
@patch('laborious.activities.activities.OPC')
@patch('laborious.activities.activities.Gates')
async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_storage_init):
@patch('laborious.activities.activities.ModelMetrics')
@patch('laborious.activities.activities.API')
async def test_shutdown(
mock_api_init,
mock_model_metrics_init,
mock_gates_init,
mock_opc_init,
mock_mlflow_init,
mock_storage_init,
):
mock_opc_init.close = AsyncMock()
postgres_config = {
'host': 'localhost',
@@ -137,6 +182,12 @@ async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_s
'group_id': 'test-group',
}
pi_web_api_config = {
'base_url': 'https://test-pi-server.com',
'auth_type': 'bearer',
'auth_token': 'test_token',
}
logger = MagicMock()
notification_handler = MagicMock()
@@ -145,6 +196,7 @@ async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_s
mlflow_config=mlflow_config,
minio_config=minio_config,
opc_config=opc_config,
pi_web_api_config=pi_web_api_config,
logger=logger,
notification_handler=notification_handler,
)
@@ -154,3 +206,5 @@ async def test_shutdown(mock_gates_init, mock_opc_init, mock_mlflow_init, mock_s
mock_storage_init.close.assert_called_once()
mock_mlflow_init.close.assert_called_once()
mock_gates_init.close.assert_called_once()
mock_model_metrics_init.close.assert_called_once()
mock_api_init.close.assert_called_once()

422
tests/laborious/activities/test_api.py Normal file
View File

@@ -0,0 +1,422 @@
from unittest.mock import ANY, AsyncMock, MagicMock, call, patch
import pytest_asyncio
from pytest import fixture, mark
from sientia_do.notifications.models import NotificationLevel
from laborious.activities.api import API, PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
metadata = {
'metadata': {
'model_id': 'test_model',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
'schema_name': 'test_schedule',
},
}
def _create_mock_dataframe(to_dict_return=None):
"""Helper function to create a mocked DataFrame for testing."""
mock_df = MagicMock()
mock_head = MagicMock()
def get_column_values(key):
if key == 'prediction':
return MagicMock(values=[0.75])
elif key == 'prediction_confidence':
return MagicMock(values=[0.95])
else:
return MagicMock(values=['2024-01-01T00:00:00+00:00'])
mock_head.__getitem__.side_effect = get_column_values
mock_df.head.return_value = mock_head
if to_dict_return is None:
to_dict_return = {
'prediction': [0.75],
'prediction_confidence': [0.95],
'timestamp': ['2024-01-01T00:00:00+00:00'],
}
mock_df.to_dict.return_value = to_dict_return
return mock_df
@fixture
def base_input_data():
"""Base input data for PI Web API tests."""
return {
**metadata,
'data': {
'prediction': [0.75],
'prediction_confidence': [0.95],
'timestamp': ['2024-01-01T00:00:00+00:00'],
},
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com/piwebapi',
'prediction_tags': {'tag1': 'web_id_1'},
'confidence_tags': {'tag2': 'web_id_2'},
},
}
def test__init__():
api = API(
base_url='https://test-pi-server.com',
auth_type='bearer',
auth_token='test_token',
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
assert api.pi_web_api_client is not None
@pytest_asyncio.fixture
@patch('laborious.activities.api.PIWebAPIClient')
def api(mock_pi_web_api_client):
mock_client = MagicMock()
mock_client.write_value = AsyncMock()
mock_client.close = MagicMock()
mock_client.base_url = 'https://test-pi-server.com'
mock_pi_web_api_client.return_value = mock_client
api_instance = API(
base_url='https://test-pi-server.com',
auth_type='bearer',
auth_token='test_token',
logger=MagicMock(),
notification_handler=MagicMock(),
metrics_controller=AsyncMock(),
)
api_instance.send_notification_async = AsyncMock()
api_instance.info = MagicMock()
api_instance.error = MagicMock()
api_instance.emit_metric = AsyncMock()
api_instance.get_core_labels = MagicMock(
return_value={
'pod_id': 'test_pod',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
}
)
return api_instance
@mark.asyncio
@patch('laborious.activities.api.DataFrame')
async def test_write_pi_web_api_data_success(mock_dataframe, api, base_input_data):
input_data = {
**base_input_data,
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com/piwebapi',
'prediction_tags': {'tag1': 'web_id_1', 'tag2': 'web_id_2'},
'confidence_tags': {'tag3': 'web_id_3', 'tag4': 'web_id_4'},
},
}
mock_dataframe.return_value = _create_mock_dataframe()
# Mock successful responses
api.pi_web_api_client.write_value.side_effect = [
[{'WebId': 'web_id_1', 'Errors': []}, {'WebId': 'web_id_2', 'Errors': []}],
[{'WebId': 'web_id_3', 'Errors': []}, {'WebId': 'web_id_4', 'Errors': []}],
]
result = await api.write_pi_web_api_data(input_data)
api.pi_web_api_client.write_value.assert_has_calls(
[
call(
web_ids=['web_id_1', 'web_id_2'],
value={
'Timestamp': '2024-01-01T00:00:00+00:00',
'Value': 0.75,
},
metadata=metadata['metadata'],
),
call(
web_ids=['web_id_3', 'web_id_4'],
value={
'Timestamp': '2024-01-01T00:00:00+00:00',
'Value': 0.95,
},
metadata=metadata['metadata'],
),
]
)
assert result == {
'prediction': [0.75],
'prediction_confidence': [0.95],
'timestamp': ['2024-01-01T00:00:00+00:00'],
}
@mark.asyncio
@patch('laborious.activities.api.DataFrame')
async def test_write_pi_web_api_data_prediction_error(mock_dataframe, api, base_input_data):
mock_dataframe.return_value = _create_mock_dataframe(
{
'prediction': [0.75],
'prediction_confidence': [PI_WEB_API_PREDICTION_ERROR_CONFIDENCE],
'timestamp': ['2024-01-01T00:00:00+00:00'],
}
)
api.pi_web_api_client.write_value.side_effect = Exception('Prediction write failed')
result = await api.write_pi_web_api_data(base_input_data)
api.send_notification_async.assert_called_once_with(
metadata=metadata['metadata'],
notification_id='WRITE_PI_WEB_API_PREDICTION_ERROR',
message="Error writing prediction data to PI Web API: Prediction write failed\n Tags: {'tag1': 'web_id_1'}",
block='write_pi_web_api_data',
level=NotificationLevel.ERROR,
attachment_content=ANY,
)
assert result['prediction_confidence'][0] == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
assert api.pi_web_api_client.write_value.call_count == 1
@mark.asyncio
@patch('laborious.activities.api.DataFrame')
async def test_write_pi_web_api_data_confidence_error(mock_dataframe, api, base_input_data):
mock_dataframe.return_value = _create_mock_dataframe()
# First call succeeds, second fails
api.pi_web_api_client.write_value.side_effect = [
[{'WebId': 'web_id_1', 'Errors': []}],
Exception('Confidence write failed'),
]
result = await api.write_pi_web_api_data(base_input_data)
api.send_notification_async.assert_called_once_with(
metadata=metadata['metadata'],
notification_id='WRITE_PI_WEB_API_CONFIDENCE_ERROR',
message="Error writing confidence data to PI Web API: Confidence write failed\n Tags: {'tag2': 'web_id_2'}",
block='write_pi_web_api_data',
level=NotificationLevel.ERROR,
attachment_content=ANY,
)
assert result == {
'prediction': [0.75],
'prediction_confidence': [0.95],
'timestamp': ['2024-01-01T00:00:00+00:00'],
}
assert api.pi_web_api_client.write_value.call_count == 2
@mark.asyncio
@patch('laborious.activities.api.DataFrame')
async def test_write_pi_web_api_data_empty_tags(mock_dataframe, api, base_input_data):
input_data = {
**base_input_data,
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com/piwebapi',
'prediction_tags': {},
'confidence_tags': {},
},
}
mock_dataframe.return_value = _create_mock_dataframe()
# Mock empty responses
api.pi_web_api_client.write_value.side_effect = [
[],
[],
]
result = await api.write_pi_web_api_data(input_data)
api.pi_web_api_client.write_value.assert_has_calls(
[
call(
web_ids=[],
value={
'Timestamp': '2024-01-01T00:00:00+00:00',
'Value': 0.75,
},
metadata=metadata['metadata'],
),
call(
web_ids=[],
value={
'Timestamp': '2024-01-01T00:00:00+00:00',
'Value': 0.95,
},
metadata=metadata['metadata'],
),
]
)
assert result == {
'prediction': [0.75],
'prediction_confidence': [0.95],
'timestamp': ['2024-01-01T00:00:00+00:00'],
}
@mark.asyncio
async def test_close(api):
api.close()
api.pi_web_api_client.close.assert_called_once()
@mark.asyncio
async def test_process_pi_web_api_response_success(api):
"""Test successful processing of PI Web API response with all tags written."""
response_data = [
{'WebId': 'web_id_1', 'Errors': []},
{'WebId': 'web_id_2', 'Errors': []},
]
tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
core_labels = {
'pod_id': 'test_pod',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
}
confidence, message = await api.process_pi_web_api_response(
response_data=response_data,
tags=tags,
core_labels=core_labels,
metadata=metadata['metadata'],
)
assert confidence == 0
assert message == ''
assert api.emit_metric.call_count == 2
# Verify that emit_metric was called with correct tags structure
call_args_list = api.emit_metric.call_args_list
assert len(call_args_list) == 2
# Check that all calls include core_labels and tag_name
for call_args in call_args_list:
assert 'tag_name' in call_args.kwargs['tags']
assert call_args.kwargs['tags']['tag_name'] in ['tag1', 'tag2']
@mark.asyncio
async def test_process_pi_web_api_response_with_errors(api):
"""Test processing response with errors in some tags."""
response_data = [
{'WebId': 'web_id_1', 'Errors': ['Error writing tag']},
{'WebId': 'web_id_2', 'Errors': []},
]
tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
core_labels = {
'pod_id': 'test_pod',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
}
confidence, message = await api.process_pi_web_api_response(
response_data=response_data,
tags=tags,
core_labels=core_labels,
metadata=metadata['metadata'],
)
assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
assert (
message
== "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag2'] tags were written."
)
assert api.emit_metric.call_count == 2
@mark.asyncio
async def test_process_pi_web_api_response_missing_tags(api):
"""Test processing response when number of written tags doesn't match expected."""
response_data = [
{'WebId': 'web_id_1', 'Errors': []},
]
tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
core_labels = {
'pod_id': 'test_pod',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
}
confidence, message = await api.process_pi_web_api_response(
response_data=response_data,
tags=tags,
core_labels=core_labels,
metadata=metadata['metadata'],
)
assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
assert (
message
== "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag1'] tags were written."
)
api.send_notification_async.assert_called_once()
call_args = api.send_notification_async.call_args
assert call_args.kwargs['notification_id'] == 'WRITE_PI_WEB_API_PREDICTION_ERROR'
assert call_args.kwargs['level'] == NotificationLevel.ERROR
@mark.asyncio
async def test_process_pi_web_api_response_missing_webid(api):
"""Test processing response when WebId is missing in response item."""
response_data = [
{'Errors': []},
{'WebId': 'web_id_2', 'Errors': []},
]
tags = {'tag1': 'web_id_1', 'tag2': 'web_id_2'}
core_labels = {
'pod_id': 'test_pod',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
}
confidence, message = await api.process_pi_web_api_response(
response_data=response_data,
tags=tags,
core_labels=core_labels,
metadata=metadata['metadata'],
)
assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
assert (
message
== "The number of written tags does not match the number of tag names: Expected ['tag1', 'tag2'] tags, but ['tag2'] tags were written."
)
api.error.assert_any_call('The response did not contain some WebIds', metadata['metadata'])
@mark.asyncio
async def test_process_pi_web_api_response_missing_tag_name(api):
"""Test processing response when tag name is not found for WebId."""
response_data = [
{'WebId': 'unknown_web_id', 'Errors': []},
]
tags = {'tag1': 'web_id_1'}
core_labels = {
'pod_id': 'test_pod',
'model_name': 'test_model',
'workflow_name': 'test_workflow',
}
confidence, message = await api.process_pi_web_api_response(
response_data=response_data,
tags=tags,
core_labels=core_labels,
metadata=metadata['metadata'],
)
assert confidence == PI_WEB_API_PREDICTION_ERROR_CONFIDENCE
assert (
message
== "The number of written tags does not match the number of tag names: Expected ['tag1'] tags, but [] tags were written."
)
api.error.assert_any_call(
'The response did not contain the tag name for WebId unknown_web_id', metadata['metadata']
)

17
tests/laborious/utils/repository/test_model_repository.py
View File

@@ -566,6 +566,12 @@ async def test_download_model_transform(mlflow_repository):
assert result == (mlflow_repository.load_transform_model.return_value, None)
def test_detect_and_parse_datetime_index_empty(mlflow_repository):
input_data = DataFrame()
response = mlflow_repository.detect_and_parse_datetime_index(input_data, metadata['metadata'])
assert response.equals(input_data)
invalid_cases = [
(
{'value': {'2024-01-01 12:00:00': 1, 2024: 2}},
@@ -812,7 +818,14 @@ async def test_fit_models_not_df_target_name_none_and_not_in_model(
data = MagicMock()
output = await mlflow_repository.fit_models(
'model_name', data, 'latest_production_id', metadata['metadata'], 'sklearn', 'pyfunc', None
'model_name',
data,
'latest_production_id',
metadata['metadata'],
'sklearn',
False,
'pyfunc',
None,
)
mlflow_repository.download_model.assert_has_calls(
@@ -908,6 +921,7 @@ async def test_fit_models_df_target_name_not_none_and_in_model(
'latest_production_id',
metadata['metadata'],
'sklearn',
False,
'pyfunc',
'feat_1',
)
@@ -1425,6 +1439,7 @@ async def test_retrain_model(mlflow_repository):
model_name=model_name,
data=data,
transform_flavor='sklearn',
skip_transform=False,
predict_flavor='pyfunc',
target_name='target',
metadata=metadata['metadata'],

75
tests/laborious/utils/test_connectors_config.py
View File

@@ -3,9 +3,7 @@ from os import environ
from laborious.utils.connectors_config import (
build_minio_config,
build_mlflow_config,
build_mongodb_config,
build_opc_config,
build_postgres_config,
)
@@ -92,79 +90,6 @@ def test_build_opc_config_with_defaults():
assert config['1']['reconnection_interval'] == 120
def test_build_postgres_config_with_env_vars():
# Arrange
environ['POSTGRES_HOST'] = 'test-host'
environ['POSTGRES_PORT'] = '5433'
environ['POSTGRES_USER'] = 'test-user'
environ['POSTGRES_PASSWORD'] = 'test-pass'
environ['POSTGRES_DBNAME'] = 'test-db'
environ['POSTGRES_MIN_CONNECTIONS'] = '10'
environ['POSTGRES_MAX_CONNECTIONS'] = '30'
# Act
config = build_postgres_config()
# Assert
assert config['host'] == 'test-host'
assert config['port'] == 5433
assert config['user'] == 'test-user'
assert config['password'] == 'test-pass'
assert config['dbname'] == 'test-db'
assert config['min_connections'] == 10
assert config['max_connections'] == 30
def test_build_postgres_config_with_defaults():
# Arrange
environ.pop('POSTGRES_HOST', None)
environ.pop('POSTGRES_PORT', None)
environ.pop('POSTGRES_USER', None)
environ.pop('POSTGRES_PASSWORD', None)
environ.pop('POSTGRES_DBNAME', None)
environ.pop('POSTGRES_MIN_CONNECTIONS', None)
environ.pop('POSTGRES_MAX_CONNECTIONS', None)
# Act
config = build_postgres_config()
# Assert
assert config['host'] == 'localhost'
assert config['port'] == 5432
assert config['user'] == 'sientia'
assert config['password'] == 'sientia'
assert config['dbname'] == 'sientia'
assert config['min_connections'] == 5
assert config['max_connections'] == 20
def test_build_mongo_db_config_with_env_vars():
environ['MONGODB_USERNAME'] = 'sientia1'
environ['MONGODB_PASSWORD'] = 'sientia1'
environ['MONGODB_URL'] = 'localhost:27018'
environ['MONGODB_DATABASE_NAME'] = 'test_db'
environ['MONGODB_TTL_INDEX_HOURS'] = '1'
assert build_mongodb_config() == {
'connection_string': 'mongodb://sientia1:sientia1@localhost:27018',
'database_name': 'test_db',
'ttl_index_seconds': 3600,
}
def test_build_mongo_db_config_with_defaults():
environ.pop('MONGODB_USERNAME', None)
environ.pop('MONGODB_PASSWORD', None)
environ.pop('MONGODB_DATABASE_NAME', None)
environ.pop('MONGODB_URL', None)
environ.pop('MONGODB_TTL_INDEX_HOURS', None)
assert build_mongodb_config() == {
'connection_string': 'mongodb://root:wKZDbMNU1c@localhost:27018',
'database_name': 'sientia',
'ttl_index_seconds': 3600,
}
def test_build_minio_config_with_env_vars():
environ['MINIO_ENDPOINT_URL'] = 'http://test-host'
environ['MINIO_ACCESS_KEY'] = 'test-key'

280
tests/laborious/workflows/subworkflows/test_format_and_export_prediction.py
View File

@@ -375,3 +375,283 @@ async def test_run_default_path_flag(workflow_mock, format_and_export_prediction
assert workflow_mock.execute_activity_method.call_count == 3
assert workflow_mock.execute_local_activity_method.call_count == 1
@mark.asyncio
@patch(
'laborious.workflows.sub_workflows.format_and_export_prediction.workflow',
new_callable=AsyncMock,
)
async def test_run_none_path_flag_with_pi_web_api(workflow_mock, format_and_export_prediction):
input_data = {
'metadata': metadata,
'path_flag': None,
'data': {'test': 'data'},
'timestamp': '2021-01-01',
'model_id': 1,
'prediction_confidence': 0,
'schema': 'test_schema',
'table_name': 'test_table',
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com',
'prediction_tags': {},
'confidence_tags': {},
},
'prediction_store_policy': 'erl:1',
}
pi_web_api_data = MagicMock()
workflow_mock.execute_activity_method.side_effect = [
pi_web_api_data, # write_pi_web_api_data
MagicMock(), # export_data_to_postgres
MagicMock(), # write_metrics
]
await format_and_export_prediction.run(input_data)
workflow_mock.execute_local_activity_method.assert_has_calls(
[
call(
Activities.format_prediction,
{
'data': input_data['data'],
'timestamp': input_data['timestamp'],
'model_id': input_data['model_id'],
'prediction_confidence': input_data['prediction_confidence'],
'prediction_store_policy': input_data['prediction_store_policy'],
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.write_pi_web_api_data,
{
'pi_web_api_output_config': input_data['pi_web_api_output_config'],
'data': workflow_mock.execute_local_activity_method.return_value,
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.export_data_to_postgres,
{
'schema': input_data['schema'],
'table_name': input_data['table_name'],
'data': pi_web_api_data,
'timestamp_conversion': {
'column': 'timestamp',
'format': DATETIME_FORMAT_WITH_TZ,
},
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.write_metrics,
{
**metadata,
'prediction': pi_web_api_data,
'opc_metrics': {},
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
assert workflow_mock.execute_activity_method.call_count == 3
assert workflow_mock.execute_local_activity_method.call_count == 1
@mark.asyncio
@patch(
'laborious.workflows.sub_workflows.format_and_export_prediction.workflow',
new_callable=AsyncMock,
)
async def test_run_none_path_flag_with_pi_web_api_and_opc(
workflow_mock, format_and_export_prediction
):
input_data = {
'metadata': metadata,
'path_flag': None,
'data': {'test': 'data'},
'timestamp': '2021-01-01',
'model_id': 1,
'prediction_confidence': 0,
'schema': 'test_schema',
'table_name': 'test_table',
'opc_output_config': {'test': 'config'},
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com',
'prediction_tags': {},
'confidence_tags': {},
},
'prediction_store_policy': 'erl:1',
}
prediction_data = MagicMock()
pi_web_api_data = MagicMock()
opc_metrics = MagicMock()
workflow_mock.execute_activity_method.side_effect = [
pi_web_api_data, # write_pi_web_api_data
(prediction_data, opc_metrics), # write_opc_data
MagicMock(), # export_data_to_postgres
MagicMock(), # write_metrics
]
await format_and_export_prediction.run(input_data)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.write_pi_web_api_data,
{
'pi_web_api_output_config': input_data['pi_web_api_output_config'],
'data': workflow_mock.execute_local_activity_method.return_value,
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
),
call(
Activities.write_opc_data,
{
'opc_output_config': input_data['opc_output_config'],
'data': pi_web_api_data,
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
),
]
)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.export_data_to_postgres,
{
'schema': input_data['schema'],
'table_name': input_data['table_name'],
'data': prediction_data,
'timestamp_conversion': {
'column': 'timestamp',
'format': DATETIME_FORMAT_WITH_TZ,
},
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.write_metrics,
{
**metadata,
'prediction': prediction_data,
'opc_metrics': opc_metrics,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
assert workflow_mock.execute_activity_method.call_count == 4
assert workflow_mock.execute_local_activity_method.call_count == 1
@mark.asyncio
@patch(
'laborious.workflows.sub_workflows.format_and_export_prediction.workflow',
new_callable=AsyncMock,
)
async def test_run_default_path_flag_with_pi_web_api(workflow_mock, format_and_export_prediction):
input_data = {
'metadata': metadata,
'path_flag': 'default',
'data': {'test': 'data'},
'timestamp': '2021-01-01',
'model_id': 1,
'prediction_confidence': 0,
'schema': 'test_schema',
'table_name': 'test_table',
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com',
'prediction_tags': {},
'confidence_tags': {},
},
'comment': 'test_comment',
}
pi_web_api_data = MagicMock()
workflow_mock.execute_activity_method.side_effect = [
pi_web_api_data, # write_pi_web_api_data
MagicMock(), # export_data_to_postgres
MagicMock(), # write_metrics
]
await format_and_export_prediction.run(input_data)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.write_pi_web_api_data,
{
'pi_web_api_output_config': input_data['pi_web_api_output_config'],
'data': workflow_mock.execute_local_activity_method.return_value,
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
workflow_mock.execute_activity_method.assert_has_calls(
[
call(
Activities.export_data_to_postgres,
{
'schema': input_data['schema'],
'table_name': input_data['table_name'],
'data': pi_web_api_data,
'timestamp_conversion': {
'column': 'timestamp',
'format': DATETIME_FORMAT_WITH_TZ,
},
**metadata,
},
retry_policy=ANY,
start_to_close_timeout=ANY,
)
]
)
assert workflow_mock.execute_activity_method.call_count == 3
assert workflow_mock.execute_local_activity_method.call_count == 1

17
tests/laborious/workflows/subworkflows/test_prediction_process.py
View File

@@ -40,6 +40,7 @@ async def test_run(workflow_mock, prediction_process):
'model_config': {'retention': '30'},
'path_priority': ['continue', 'repeat', 'stop'],
'opc_output_config': {'test': 'config'},
'pi_web_api_output_config': {'test': 'config'},
'prediction_store_policy': 'lts:1',
}
@@ -183,6 +184,7 @@ async def test_run(workflow_mock, prediction_process):
'model_name': 'test_model_name',
'model_config': input_data['model_config'],
'opc_output_config': input_data['opc_output_config'],
'pi_web_api_output_config': input_data['pi_web_api_output_config'],
'schema': input_data['schema'],
'table_name': input_data['table_name'],
'transform_table_name': input_data['transform_table_name'],
@@ -729,6 +731,11 @@ async def test_path_flag_handler_continue(workflow_mock, prediction_process):
'model_name': model_name,
'model_config': model_config,
'opc_output_config': {'test': 'config'},
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com',
'prediction_tags': {},
'confidence_tags': {},
},
'prediction_store_policy': prediction_store_policy,
},
confidence,
@@ -755,6 +762,11 @@ async def test_path_flag_handler_continue(workflow_mock, prediction_process):
'transform_table_name': 'test_transform_table',
'comment': 'Prediction Process',
'opc_output_config': {'test': 'config'},
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com',
'prediction_tags': {},
'confidence_tags': {},
},
'prediction_store_policy': prediction_store_policy,
},
)
@@ -788,6 +800,11 @@ async def test_path_flag_handler_unknown(workflow_mock, prediction_process):
'model_name': model_name,
'model_config': model_config,
'opc_output_config': {'test': 'config'},
'pi_web_api_output_config': {
'endpoint': 'https://test-pi-server.com',
'prediction_tags': {},
'confidence_tags': {},
},
'prediction_store_policy': prediction_store_policy,
},
confidence,

3
tests/laborious/workflows/test_predictions_batch.py
View File

@@ -34,6 +34,7 @@ async def test_run(workflow_mock: AsyncMock, predictions_batch: PredictionsBatch
'table_name': 'test_table',
'transform_table_name': 'test_transform_table',
'opc_output_config': 'test_opc_output_config',
'pi_web_api_output_config': 'test_pi_web_api_output_config',
'datetime_columns': ['timestamp', 'created_at'],
'prediction_store_policy': 'erl:1',
'model_config': {'retention': '30'},
@@ -73,7 +74,9 @@ async def test_run(workflow_mock: AsyncMock, predictions_batch: PredictionsBatch
'model_config': input_data.get('model_config', {}),
'path_priority': input_data.get('path_priority', ['STOP', 'CONTINUE', 'REPEAT']),
'opc_output_config': input_data.get('opc_output_config', {}),
'pi_web_api_output_config': input_data.get('pi_web_api_output_config', {}),
'prediction_store_policy': input_data.get('prediction_store_policy', 'lts:1'),
'save_transform': input_data.get('save_transform', True),
}
workflow_mock.execute_child_workflow.assert_has_calls(

4
validate.sh
View File

@@ -70,14 +70,14 @@ FAILED_STEPS=()
# Step 1: Code Formatting Check (Ruff)
# - default: check only
# - --fix: write changes
if ! run_step "1. Code Formatting (Ruff)" "if \$FIX_MODE; then ruff format laborious/ tests/; else ruff format --check laborious/ tests/; fi"; then
if ! run_step "1. Code Formatting (Ruff)" "if \$FIX_MODE; then ruff format laborious/ tests/; else ruff format --check laborious/ tests/ e2e/; fi"; then
FAILED_STEPS+=("Code Formatting")
fi
# Step 2: Linting (Ruff)
# - default: check only
# - --fix: apply autofixes
if ! run_step "2. Code Linting (Ruff)" "if \$FIX_MODE; then ruff check --fix laborious/ tests/; else ruff check laborious/ tests/; fi"; then
if ! run_step "2. Code Linting (Ruff)" "if \$FIX_MODE; then ruff check --fix laborious/ tests/; else ruff check laborious/ tests/ e2e/; fi"; then
FAILED_STEPS+=("Linting")
fi

43
values.yaml
View File

@@ -3,17 +3,17 @@
# Declare variables to be passed into your templates.
# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
replicaCount: 2
replicaCount: 1
# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
image:
repository: aignosi.azurecr.io/sientia-module-courier
repository: aignosi.azurecr.io/sientia-module
# This sets the pull policy for images.
pullPolicy: Always
# Overrides the image tag whose default is the chart appVersion.
tag: "1.1.2"
0# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
imagePullSecrets:
- name: docker-hub-secret
# This is to override the chart name.
@@ -62,24 +62,32 @@ resources:
cpu: 1000m # 1 CPU core
memory: 2Gi # 2 GB memory
# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
livenessProbe:
exec:
command:
- sh
- -c
- pgrep -f "laborious.worker.worker"
initialDelaySeconds: 20
periodSeconds: 30
- |
curl -sf http://localhost:9090/metrics | grep -q '^app_up{.*} 1'
initialDelaySeconds: 660
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command:
- sh
- -c
- pgrep -f "laborious.worker.worker"
initialDelaySeconds: 10
periodSeconds: 15
- |
curl -sf http://localhost:9090/metrics | grep -q '^app_up{.*} 1'
initialDelaySeconds: 600
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 2
# This section is for setting up autoscaling more information can be found here: https://kubernetes.io/docs/concepts/workloads/autoscaling/
@@ -149,7 +157,7 @@ env:
- name: GITHUB_REPO_URL
value: "git@github.com:Aignosi/sientia-dataops-laborious_temporal.git"
- name: GITHUB_BRANCH
value: "feature/SIENTIAPDE-1273"
value: "feature/SIENTIAPDE-1478"
- name: PYTHON_APP
value: "laborious.worker.worker"
@@ -187,8 +195,6 @@ env:
- name: OPC_URL
value: "opc.tcp://sientia-opc-simulator-opc.sientia.svc.cluster.local:4840"
- name: KAFKA_BOOTSTRAP_SERVERS
value: "kafka.kafka.svc.cluster.local:9092"
- name: LOG_LEVEL
value: "DEBUG"
@@ -226,6 +232,19 @@ env:
- name: MINIO_DEFAULT_BUCKET
value: "sientia"
- name: PI_WEB_API_BASE_URL
value: "https://pivision.votorantimcimentos.com/piwebapi"
- name: PI_WEB_API_AUTH_TYPE
value: "basic"
- name: PI_WEB_API_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: pi-web-api-auth-token
key: token
- name: PYPI_SERVER
value: "http://library-distribution-server.library.svc.cluster.local:5000"
ssh:
enabled: true
secretName: git-ssh-key-sientia-laborious-worker