9bcf882d00
Update .gitignore to include git_log and enhance README.md with instructions for generating a git log summary.
102 lines
3.7 KiB
Markdown
102 lines
3.7 KiB
Markdown
# Sientia DataOps Laborious
|
|
|
|
The Sientia DataOps Laborious is a Temporal-based workflow application that handles batch predictions and data processing for industrial data. It integrates with MLFlow for model management, PostgreSQL for data storage, and OPC for real-time data output. The module is designed to process data in a reliable and scalable manner using Temporal.io's workflow orchestration capabilities. It's get data from Scouter sinks, process it, make predictions using MLFlow models and generates metrics for the predictions.
|
|
|
|
## Key Features
|
|
|
|
- Batch predictions using MLFlow models
|
|
- Data transformation and preprocessing
|
|
- Workflow orchestration using Temporal.io
|
|
- Integration with PostgreSQL for data storage
|
|
- OPC integration for real-time data output
|
|
- Comprehensive error handling and notifications
|
|
- Configurable data filters and quality gates
|
|
- Scalable deployment architecture
|
|
|
|
## Workflows
|
|
|
|
### Predictions Batch
|
|
The main workflow that orchestrates batch predictions. Steps:
|
|
|
|
- prepare_activity: Prepares the activity with schedule and model information
|
|
- load_custom_query: Loads data using a custom query
|
|
- prediction_process: Executes the prediction process using the Prediction Process sub-workflow
|
|
|
|
#### Workflow inputs:
|
|
|
|
- `schedule_name`: The schedule name of the activity
|
|
- `model_name`: The model name of the activity
|
|
- `model_id`: The model id of the activity
|
|
- `query`: The custom query to load data
|
|
- `schema`: The schema of the data
|
|
- `table_name`: The name of the table to process
|
|
- `input_filters`: The filters to be applied during prediction
|
|
- `mlflow_transform_filters`: The filters to be applied during prediction
|
|
- `mlflow_predict_filters`: The filters to be applied during prediction
|
|
- `model_retention`: The model retention period in minutes
|
|
- `path_priority`: The path priority
|
|
|
|
|
|
### Prediction Process
|
|
Sub-workflow that handles individual prediction processing:
|
|
|
|
- get_last_timestamp: Gets the last timestamp of the data
|
|
- input_gate: Filters input data based on configured rules
|
|
- repeat_last_prediction: Repeats the last prediction if the data is empty
|
|
- request_transform: Makes predictions using MLFlow models
|
|
- mlflow_response_gate: Handles prediction or transform responses and filters
|
|
- mlflow_content_gate: Filters transform responses based on configured rules
|
|
- request_predict: Makes predictions using MLFlow models
|
|
- format_and_export_prediction: Formats and exports predictions using the
|
|
Format and Export Prediction sub-workflow
|
|
|
|
### Format and Export Prediction
|
|
Sub-workflow that handles prediction formatting and export:
|
|
|
|
- format_prediction: Formats prediction data if path flag is None
|
|
- format_default_prediction: Formats default prediction data if path flag is not None
|
|
- export_to_postgres: Exports formatted predictions to PostgreSQL
|
|
- write_to_opc: Writes predictions to OPC server
|
|
|
|
## Environment variables
|
|
|
|
- `POSTGRES_HOST`
|
|
- `POSTGRES_PORT`
|
|
- `POSTGRES_USER`
|
|
- `POSTGRES_PASSWORD`
|
|
- `POSTGRES_DBNAME`
|
|
- `POSTGRES_MIN_CONNECTIONS`
|
|
- `POSTGRES_MAX_CONNECTIONS`
|
|
|
|
- `MLFLOW_HOST`
|
|
- `MLFLOW_PORT`
|
|
- `MLFLOW_USERNAME`
|
|
- `MLFLOW_PASSWORD`
|
|
|
|
- `OPC_CONFIG` - json string containing the opc configuration for multiple opc servers
|
|
For single opc server use:
|
|
- `OPC_URL`
|
|
- `OPC_NAME`
|
|
- `OPC_SERVER_URI`
|
|
- `OPC_CERT_PATH`
|
|
- `OPC_PRIVATE_KEY_PATH`
|
|
- `OPC_SERVER_CERT_PATH`
|
|
- `OPC_RECONNECTION_INTERVAL`
|
|
|
|
- `TEMPORAL_HOST`
|
|
- `TEMPORAL_NAMESPACE`
|
|
|
|
## Application deployment
|
|
|
|
The application can be deployed using the following command:
|
|
|
|
```bash
|
|
helm upgrade --install sientia-dataops-laborious sientia/sientia-module -n sientia --create-namespace -f ./values.yaml
|
|
```
|
|
|
|
#PR shortcut
|
|
```
|
|
git log origin/main..HEAD --no-merges > git_log
|
|
```
|
|
Prompt:
|
|
Write a summary of PR changes in markdown. Be objective and direct. Write to file |