# Sientia DataOps Laborious

The Sientia DataOps Laborious is a Temporal-based workflow application that handles batch predictions and data processing for industrial data. It integrates with MLFlow for model management, PostgreSQL for data storage, and OPC for real-time data output. The module is designed to process data in a reliable and scalable manner using Temporal.io's workflow orchestration capabilities. It's get data from Scouter sinks, process it, make predictions using MLFlow models and generates metrics for the predictions.

## Key Features

- Batch predictions using MLFlow models
- Data transformation and preprocessing
- Workflow orchestration using Temporal.io
- Integration with PostgreSQL for data storage
- OPC integration for real-time data output
- Comprehensive error handling and notifications
- Configurable data filters and quality gates
- Scalable deployment architecture

## Workflows

### Predictions Batch
The main workflow that orchestrates batch predictions. Steps:

- prepare_activity: Prepares the activity with schedule and model information
- load_custom_query: Loads data using a custom query
- prediction_process: Executes the prediction process using the Prediction Process sub-workflow

#### Workflow inputs:

- `schedule_name`: The schedule name of the activity
- `model_name`: The model name of the activity
- `model_id`: The model id of the activity
- `query`: The custom query to load data
- `schema`: The schema of the data
- `table_name`: The name of the table to process
- `input_filters`: The filters to be applied during prediction
- `mlflow_transform_filters`: The filters to be applied during prediction
- `mlflow_predict_filters`: The filters to be applied during prediction
- `model_retention`: The model retention period in minutes
- `path_priority`: The path priority


### Prediction Process
Sub-workflow that handles individual prediction processing:

- get_last_timestamp: Gets the last timestamp of the data
- input_gate: Filters input data based on configured rules
- repeat_last_prediction: Repeats the last prediction if the data is empty
- request_transform: Makes predictions using MLFlow models
- mlflow_response_gate: Handles prediction or transform responses and filters
- mlflow_content_gate: Filters transform responses based on configured rules
- request_predict: Makes predictions using MLFlow models
- format_and_export_prediction: Formats and exports predictions using the
    Format and Export Prediction sub-workflow

### Format and Export Prediction
Sub-workflow that handles prediction formatting and export:

- format_prediction: Formats prediction data if path flag is None
- format_default_prediction: Formats default prediction data if path flag is not None
- export_to_postgres: Exports formatted predictions to PostgreSQL
- write_to_opc: Writes predictions to OPC server

## Environment variables

- `POSTGRES_HOST`
- `POSTGRES_PORT`
- `POSTGRES_USER`
- `POSTGRES_PASSWORD`
- `POSTGRES_DBNAME`
- `POSTGRES_MIN_CONNECTIONS`
- `POSTGRES_MAX_CONNECTIONS`

- `MLFLOW_HOST`
- `MLFLOW_PORT`
- `MLFLOW_USERNAME`
- `MLFLOW_PASSWORD`

- `OPC_CONFIG` - json string containing the opc configuration for multiple opc servers
For single opc server use:
- `OPC_URL`
- `OPC_NAME`
- `OPC_SERVER_URI`
- `OPC_CERT_PATH`
- `OPC_PRIVATE_KEY_PATH`
- `OPC_SERVER_CERT_PATH`
- `OPC_RECONNECTION_INTERVAL`

- `TEMPORAL_HOST`
- `TEMPORAL_NAMESPACE`

## Application deployment

The application can be deployed using the following command:

```bash
helm upgrade --install sientia-dataops-laborious sientia/sientia-module -n sientia --create-namespace -f ./values.yaml
```

#PR shortcut
```
git log origin/main..HEAD --no-merges > git_log
```
Prompt:
Write a summary of PR changes in markdown. Be objective and direct. Write to file