SIENTIAPDE-1712
Enhance README and Implement Drift Detection and Metrics Workflows - Added new sections in README for Drift Workflow and Simple Metrics Workflow, detailing their execution flows and functionalities. - Introduced `drift.py` for data drift detection, comparing current data against reference datasets. - Added `simple_metrics.py` for calculating regression metrics (RMSE, MSE, MAE, R²). - Updated `values.yaml` to include configuration for MinIO retention hours and offload threshold. - Refactored `minio_dataframe_payload.py` to use the new offload threshold environment variable. - Adjusted tests to reflect changes in environment variable handling for MinIO offload threshold.
This commit is contained in:
217
README.md
217
README.md
@@ -19,6 +19,8 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
|
|||||||
- [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
|
- [Prediction Process Workflow](#2-prediction-process-workflow-prediction_processpy)
|
||||||
- [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
|
- [Format and Export Prediction Workflow](#3-format-and-export-prediction-workflow-format_and_export_predictionpy)
|
||||||
- [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
|
- [Minimal Retrain Workflow](#4-minimal-retrain-workflow-minimal_retrainpy)
|
||||||
|
- [Drift Workflow](#5-drift-workflow-driftpy)
|
||||||
|
- [Simple Metrics Workflow](#6-simple-metrics-workflow-simple_metricspy)
|
||||||
- [Installation & Setup](#installation--setup)
|
- [Installation & Setup](#installation--setup)
|
||||||
- [Prerequisites](#prerequisites)
|
- [Prerequisites](#prerequisites)
|
||||||
- [Environment Setup](#environment-setup)
|
- [Environment Setup](#environment-setup)
|
||||||
@@ -77,13 +79,16 @@ A comprehensive, Temporal-based ML orchestration system for industrial data proc
|
|||||||
### Advanced Capabilities
|
### Advanced Capabilities
|
||||||
- **Incremental Data Processing**: Timestamp-based loading to avoid reprocessing
|
- **Incremental Data Processing**: Timestamp-based loading to avoid reprocessing
|
||||||
- **Configurable Data Retention**: Model retention policies with automatic cleanup
|
- **Configurable Data Retention**: Model retention policies with automatic cleanup
|
||||||
|
- **MinIO Payload Offload**: Automatic offload of large DataFrames to MinIO with retention cleanup
|
||||||
|
- **Data Drift Detection**: Univariate and multivariate drift monitoring against reference data
|
||||||
|
- **Regression Metrics**: Automated RMSE, MSE, MAE, R² calculation and export
|
||||||
- **Notification System**: Integrated alerting via MongoDB
|
- **Notification System**: Integrated alerting via MongoDB
|
||||||
- **Scalable Architecture**: Kubernetes-ready with horizontal scaling
|
- **Scalable Architecture**: Kubernetes-ready with horizontal scaling
|
||||||
- **Model Retraining**: Automated retraining workflows with production model updates
|
- **Model Retraining**: Automated retraining workflows with production model updates
|
||||||
|
|
||||||
### Development & Quality Assurance
|
### Development & Quality Assurance
|
||||||
- **Code Quality Tools**: Ruff (lint/format), mypy (types), Bandit (security)
|
- **Code Quality Tools**: Ruff (lint/format), mypy (types), Bandit (security)
|
||||||
- **Automated Validation**: `validate.sh` and CI quality gates
|
- **Automated Validation**: CI quality gates and individual tool commands
|
||||||
- **Comprehensive Testing**: pytest with async support and high coverage
|
- **Comprehensive Testing**: pytest with async support and high coverage
|
||||||
- **Type Safety**: Static type checking with mypy
|
- **Type Safety**: Static type checking with mypy
|
||||||
- **Coverage Visualization**: Coverage Gutters integration
|
- **Coverage Visualization**: Coverage Gutters integration
|
||||||
@@ -129,6 +134,8 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
|
|||||||
- `sub_workflows/prediction_process.py`: Core prediction pipeline
|
- `sub_workflows/prediction_process.py`: Core prediction pipeline
|
||||||
- `sub_workflows/format_and_export_prediction.py`: Formatting and export
|
- `sub_workflows/format_and_export_prediction.py`: Formatting and export
|
||||||
- `minimal_retrain.py`: Automated model retraining and production update
|
- `minimal_retrain.py`: Automated model retraining and production update
|
||||||
|
- `drift.py`: Data drift detection and monitoring
|
||||||
|
- `simple_metrics.py`: Regression metrics calculation (RMSE, MSE, MAE, R²)
|
||||||
|
|
||||||
#### **Activities (`laborious/activities/`)**
|
#### **Activities (`laborious/activities/`)**
|
||||||
- `gates.py`: Data quality validation, filtering, and data formatting operations
|
- `gates.py`: Data quality validation, filtering, and data formatting operations
|
||||||
@@ -139,16 +146,26 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
|
|||||||
- MLFlow model transformation and prediction
|
- MLFlow model transformation and prediction
|
||||||
- Model retraining and production updates
|
- Model retraining and production updates
|
||||||
- Reference data retrieval from MLflow Model Registry
|
- Reference data retrieval from MLflow Model Registry
|
||||||
|
- `storage.py`: PostgreSQL queries and MinIO-aware data loading
|
||||||
|
- `load_query_with_minio_offload`: SQL load with automatic MinIO offload
|
||||||
|
- `export_payload_to_postgres`: Resolve MinIO payloads and export to Postgres
|
||||||
|
- `cleanup_minio_objects_expired`: Retention-based MinIO object cleanup
|
||||||
|
- `query_to_minio`: Legacy parquet upload for retraining data
|
||||||
|
- `model_metrics.py`: Drift detection and regression metrics
|
||||||
|
- Univariate and multivariate drift calculation
|
||||||
|
- Simple metrics (RMSE, MSE, MAE, R²)
|
||||||
- `opc.py`: OPC UA export to industrial systems (optional)
|
- `opc.py`: OPC UA export to industrial systems (optional)
|
||||||
- `api.py`: PI Web API export operations (optional)
|
- `api.py`: PI Web API export operations (optional)
|
||||||
- Prediction and confidence data writing to PI Web API
|
- Prediction and confidence data writing to PI Web API
|
||||||
- Error handling and notification integration
|
- Error handling and notification integration
|
||||||
- `activities.py`: Aggregates activity interfaces
|
- `activities.py`: Aggregates all activity interfaces (Storage, MLFlow, Gates, OPC, ModelMetrics, API)
|
||||||
|
|
||||||
#### **Data Services (`laborious/utils/`)**
|
#### **Data Services (`laborious/utils/`)**
|
||||||
- `connectors_config.py`: Env-driven configuration builders
|
- `connectors_config.py`: Env-driven configuration builders
|
||||||
|
- `models/minio_dataframe_payload.py`: MinIO-offloaded DataFrame payload model
|
||||||
- `repository/model_repository.py`: MLFlow operations and retraining
|
- `repository/model_repository.py`: MLFlow operations and retraining
|
||||||
- `repository/opc_repository.py`: OPC communication and writes
|
- `repository/opc_repository.py`: OPC communication and writes
|
||||||
|
- `repository/minio_manager.py`: MinIO object storage operations
|
||||||
- `filters/conditional_filters.py` and `filters/mlflow_filters.py`
|
- `filters/conditional_filters.py` and `filters/mlflow_filters.py`
|
||||||
|
|
||||||
### Data Flow Architecture
|
### Data Flow Architecture
|
||||||
@@ -167,6 +184,18 @@ Training Data → Model Retraining → Quality Validation →
|
|||||||
Production Update → Notification & Monitoring
|
Production Update → Notification & Monitoring
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### **3. Drift Detection Pipeline**
|
||||||
|
```
|
||||||
|
Target Data (PostgreSQL) + Reference Data (MLFlow) →
|
||||||
|
Drift Calculation (univariate + multivariate) → PostgreSQL Export
|
||||||
|
```
|
||||||
|
|
||||||
|
#### **4. Simple Metrics Pipeline**
|
||||||
|
```
|
||||||
|
Predictions + Targets (PostgreSQL JOIN) →
|
||||||
|
Metrics Calculation (RMSE, MSE, MAE, R²) → PostgreSQL Export
|
||||||
|
```
|
||||||
|
|
||||||
### Security Architecture
|
### Security Architecture
|
||||||
|
|
||||||
#### **Authentication & Authorization**
|
#### **Authentication & Authorization**
|
||||||
@@ -262,20 +291,21 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M
|
|||||||
- **Prediction Export**: Delegates prediction formatting and export operations
|
- **Prediction Export**: Delegates prediction formatting and export operations
|
||||||
|
|
||||||
#### Execution Flow
|
#### Execution Flow
|
||||||
1. **Timestamp Retrieval**: Gets the last processed timestamp for incremental processing
|
1. **Input Data Gate**: Applies configured filters for data quality validation
|
||||||
2. **Input Data Gate**: Applies configured filters for data quality validation
|
2. **Path Decision**: Determines processing path based on filter results
|
||||||
3. **Path Decision**: Determines processing path based on filter results
|
3. **MLFlow Transform**: Requests data transformation using MLFlow models
|
||||||
4. **MLFlow Transform**: Requests data transformation using MLFlow models
|
4. **Response Validation**: Filters transform responses for quality assurance
|
||||||
5. **Response Validation**: Filters transform responses for quality assurance
|
5. **Content Validation**: Filters transformed data content for quality check
|
||||||
6. **MLFlow Prediction**: Executes prediction using transformed data
|
6. **MLFlow Prediction**: Executes prediction using transformed data
|
||||||
7. **Content Validation**: Filters prediction responses for final quality check
|
7. **Prediction Response Validation**: Filters prediction responses for final quality check
|
||||||
8. **Export Delegation**: Delegates to FormatAndExportPrediction workflow
|
8. **Export Delegation**: Delegates to FormatAndExportPrediction workflow
|
||||||
|
9. **MinIO Cleanup**: Cleans up expired offloaded payloads (if any, in `finally` block)
|
||||||
|
|
||||||
#### Key Features
|
#### Key Features
|
||||||
- **Configurable Quality Gates**: Multiple filter types with policy-based configuration
|
- **Configurable Quality Gates**: Multiple filter types with policy-based configuration
|
||||||
- **Flexible Path Handling**: Configurable decision paths (STOP, CONTINUE, REPEAT)
|
- **Flexible Path Handling**: Configurable decision paths (STOP, CONTINUE, REPEAT)
|
||||||
- **MLFlow Integration**: Comprehensive model management and inference
|
- **MLFlow Integration**: Comprehensive model management and inference
|
||||||
- **Incremental Processing**: Timestamp-based data processing optimization
|
- **MinIO Cleanup**: Automatic retention-based cleanup of offloaded payloads
|
||||||
- **Comprehensive Monitoring**: Detailed metrics and error reporting
|
- **Comprehensive Monitoring**: Detailed metrics and error reporting
|
||||||
|
|
||||||
#### Input Parameters
|
#### Input Parameters
|
||||||
@@ -320,12 +350,12 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M
|
|||||||
#### Architecture Diagram
|
#### Architecture Diagram
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
A[1. get_last_timestamp] --> B[2. input_gate] --> C[3. request_transform] --> D[4. mlflow_response_gate] --> E[5. mlflow_content_gate] --> F[6. request_predict] --> G[7. mlflow_response_gate] --> H[8. format_and_export_prediction🔃]
|
A[1. input_gate] --> B[2. request_transform] --> C[3. mlflow_response_gate] --> D[4. mlflow_content_gate] --> E[5. request_predict] --> F[6. mlflow_response_gate] --> G[7. format_and_export_prediction🔃]
|
||||||
|
G --> H[8. cleanup_minio_objects_expired]
|
||||||
|
|
||||||
A -.-> Redis[(Redis)]
|
B -.-> MLFlow[MLFlow]
|
||||||
C -.-> MLFlow[MLFlow]
|
E -.-> MLFlow[MLFlow]
|
||||||
F -.-> MLFlow[MLFlow]
|
H -.-> MinIO[(MinIO)]
|
||||||
G -.-> Filters[MLFlow Filters]
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)
|
### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)
|
||||||
@@ -403,6 +433,76 @@ flowchart LR
|
|||||||
D -.-> PostgreSQL[(PostgreSQL)]
|
D -.-> PostgreSQL[(PostgreSQL)]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 5. Drift Workflow (`drift.py`)
|
||||||
|
|
||||||
|
The **Drift** workflow detects data drift by comparing current data against a reference dataset from the MLflow Model Registry.
|
||||||
|
|
||||||
|
#### Execution Flow
|
||||||
|
1. **Data Loading**: Loads target data and reference data in parallel
|
||||||
|
2. **Drift Calculation**: Calculates univariate and multivariate drift metrics
|
||||||
|
3. **Data Export**: Exports drift metrics to PostgreSQL
|
||||||
|
|
||||||
|
#### Architecture Diagram
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
A[1. load_custom_query] --> C[3. calculate_drift] --> D[4. export_data_to_postgres]
|
||||||
|
B[2. get_reference_data] --> C
|
||||||
|
|
||||||
|
A -.-> Database[(Database)]
|
||||||
|
B -.-> MLFlow[MLFlow]
|
||||||
|
D -.-> PostgreSQL[(PostgreSQL)]
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Input Parameters
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schedule_name": "hourly_drift",
|
||||||
|
"model_name": "temperature_model",
|
||||||
|
"model_id": "temp_001",
|
||||||
|
"schema": "sientia_data",
|
||||||
|
"source_table_name": "laborious_data",
|
||||||
|
"target_table_name": "drift_metrics",
|
||||||
|
"interval": 60,
|
||||||
|
"model_config": { "target": "temperature" },
|
||||||
|
"drift_metrics": ["kolmogorov_smirnov", "jensen_shannon", "wasserstein"],
|
||||||
|
"chunk_period": "min"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Simple Metrics Workflow (`simple_metrics.py`)
|
||||||
|
|
||||||
|
The **SimpleMetrics** workflow calculates regression metrics (RMSE, MSE, MAE, R²) by comparing predictions against actual target values.
|
||||||
|
|
||||||
|
#### Execution Flow
|
||||||
|
1. **Data Loading**: Loads prediction vs target data via a JOIN query
|
||||||
|
2. **Metrics Calculation**: Calculates configured regression metrics
|
||||||
|
3. **Data Export**: Exports metrics to PostgreSQL
|
||||||
|
|
||||||
|
#### Architecture Diagram
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
A[1. load_custom_query] --> B[2. calculate_simple_metrics] --> C[3. export_data_to_postgres]
|
||||||
|
|
||||||
|
A -.-> Database[(Database)]
|
||||||
|
C -.-> PostgreSQL[(PostgreSQL)]
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Input Parameters
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schedule_name": "hourly_metrics",
|
||||||
|
"model_name": "temperature_model",
|
||||||
|
"model_id": "temp_001",
|
||||||
|
"schema": "sientia_data",
|
||||||
|
"predictions_table_name": "predictions",
|
||||||
|
"data_table_name": "laborious_data",
|
||||||
|
"target_table_name": "simple_metrics",
|
||||||
|
"interval_minutes": 60,
|
||||||
|
"model_config": { "target": "temperature" },
|
||||||
|
"metrics": ["rmse", "mse", "mae", "r2"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## 📋 Prerequisites
|
## 📋 Prerequisites
|
||||||
|
|
||||||
- Python 3.11+
|
- Python 3.11+
|
||||||
@@ -527,8 +627,8 @@ pytest
|
|||||||
pytest --cov=laborious --cov-report=html
|
pytest --cov=laborious --cov-report=html
|
||||||
|
|
||||||
# Run specific test categories
|
# Run specific test categories
|
||||||
pytest tests/activities/
|
pytest tests/laborious/activities/
|
||||||
pytest tests/workflow/
|
pytest tests/laborious/workflows/
|
||||||
```
|
```
|
||||||
|
|
||||||
### Manual Application Execution
|
### Manual Application Execution
|
||||||
@@ -569,18 +669,7 @@ pip install -r requirements-dev.txt
|
|||||||
|
|
||||||
### Complete Validation
|
### Complete Validation
|
||||||
|
|
||||||
Option 1 (recommended):
|
Run each validation step individually:
|
||||||
```bash
|
|
||||||
./validate.sh
|
|
||||||
```
|
|
||||||
The script runs, in order:
|
|
||||||
1. Format check (Ruff)
|
|
||||||
2. Linting (Ruff)
|
|
||||||
3. Type checking (mypy)
|
|
||||||
4. Security analysis (Bandit)
|
|
||||||
5. Tests with coverage (pytest)
|
|
||||||
|
|
||||||
Option 2 (individual commands):
|
|
||||||
```bash
|
```bash
|
||||||
ruff format --check laborious/ tests/
|
ruff format --check laborious/ tests/
|
||||||
ruff check laborious/ tests/
|
ruff check laborious/ tests/
|
||||||
@@ -606,7 +695,7 @@ The workflow at `.github/workflows/quality-gate.yml` executes validations on eac
|
|||||||
|
|
||||||
### Best Practices
|
### Best Practices
|
||||||
|
|
||||||
- Run `./validate.sh` before committing
|
- Run all validation steps before committing
|
||||||
- Use `ruff check --watch` for continuous feedback
|
- Use `ruff check --watch` for continuous feedback
|
||||||
- Add type hints and tests for new code
|
- Add type hints and tests for new code
|
||||||
|
|
||||||
@@ -615,15 +704,34 @@ The workflow at `.github/workflows/quality-gate.yml` executes validations on eac
|
|||||||
### Test Structure
|
### Test Structure
|
||||||
```
|
```
|
||||||
tests/
|
tests/
|
||||||
├── activities/ # Activity implementation tests
|
├── conftest.py # Global fixtures and env setup
|
||||||
│ ├── test_gates.py # Data quality gates and formatting tests
|
├── laborious/
|
||||||
│ ├── test_mlflow.py # MLFlow operations and reference data tests
|
│ ├── activities/ # Activity implementation tests
|
||||||
│ └── ... # Other activity tests
|
│ │ ├── test_activities.py # Activities aggregator tests
|
||||||
├── workflows/ # Workflow orchestration tests
|
│ │ ├── test_gates.py # Data quality gates and formatting tests
|
||||||
│ └── subworkflows/ # Sub-workflow tests
|
│ │ ├── test_mlflow.py # MLFlow operations and reference data tests
|
||||||
│ └── test_format_and_export_prediction.py # Export workflow tests
|
│ │ ├── test_storage.py # Storage and MinIO offload tests
|
||||||
├── utils/ # Utility function tests
|
│ │ ├── test_model_metrics.py # Drift and simple metrics tests
|
||||||
└── integration/ # End-to-end workflow tests
|
│ │ ├── test_opc.py # OPC operations tests
|
||||||
|
│ │ └── test_api.py # PI Web API operations tests
|
||||||
|
│ ├── workflows/ # Workflow orchestration tests
|
||||||
|
│ │ ├── test_predictions_batch.py
|
||||||
|
│ │ ├── test_minimal_retrain.py
|
||||||
|
│ │ ├── test_drift.py
|
||||||
|
│ │ ├── test_simple_metrics.py
|
||||||
|
│ │ └── subworkflows/
|
||||||
|
│ │ ├── test_prediction_process.py
|
||||||
|
│ │ └── test_format_and_export_prediction.py
|
||||||
|
│ └── utils/ # Utility function tests
|
||||||
|
│ ├── test_connectors_config.py
|
||||||
|
│ ├── models/
|
||||||
|
│ │ └── test_minio_dataframe_payload.py
|
||||||
|
│ ├── filters/
|
||||||
|
│ │ ├── test_conditional_filters.py
|
||||||
|
│ │ └── test_mlflow_filters.py
|
||||||
|
│ └── repository/
|
||||||
|
│ ├── test_model_repository.py
|
||||||
|
│ └── test_opc_repository.py
|
||||||
```
|
```
|
||||||
|
|
||||||
### Test Coverage
|
### Test Coverage
|
||||||
@@ -643,8 +751,8 @@ pip install pytest pytest-cov pytest-asyncio
|
|||||||
pytest --cov=laborious --cov-report=html
|
pytest --cov=laborious --cov-report=html
|
||||||
|
|
||||||
# Run specific test modules
|
# Run specific test modules
|
||||||
pytest tests/activities/test_gates.py
|
pytest tests/laborious/activities/test_gates.py
|
||||||
pytest tests/workflow/test_predictions_batch.py
|
pytest tests/laborious/workflows/test_predictions_batch.py
|
||||||
```
|
```
|
||||||
|
|
||||||
## 📊 Monitoring and Metrics
|
## 📊 Monitoring and Metrics
|
||||||
@@ -716,8 +824,13 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
|
|||||||
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
|
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
|
||||||
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
|
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
|
||||||
| `POD_ID` | Kubernetes pod identifier | `None` | No |
|
| `POD_ID` | Kubernetes pod identifier | `None` | No |
|
||||||
| `SIENTIA_MINIO_RETENTION_HOURS` | Retention window for offloaded MinIO objects | `168` | No |
|
| `MINIO_ENDPOINT_URL` | MinIO endpoint URL | `http://localhost:9000` | Yes |
|
||||||
| `SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES` | Offload threshold for DataFrame-derived payloads | `int(1.5 * 1024 * 1024)` | No |
|
| `MINIO_ACCESS_KEY` | MinIO access key | `minioadmin` | Yes |
|
||||||
|
| `MINIO_SECRET_KEY` | MinIO secret key | `minioadmin` | Yes |
|
||||||
|
| `MINIO_REGION_NAME` | MinIO region name | `us-east-1` | No |
|
||||||
|
| `MINIO_DEFAULT_BUCKET` | Default MinIO bucket | `laborious` | No |
|
||||||
|
| `MINIO_RETENTION_HOURS` | Retention window (hours) for offloaded MinIO objects | `24` | No |
|
||||||
|
| `SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES` | Offload threshold for DataFrame-derived payloads | `int(1.5 * 1024 * 1024)` | No |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -728,7 +841,7 @@ Laborious uses MinIO to prevent Temporal workflow history from carrying very lar
|
|||||||
Whenever a payload exceeds a configurable size threshold, it is stored as a parquet file in MinIO and the workflow history only keeps a lightweight reference.
|
Whenever a payload exceeds a configurable size threshold, it is stored as a parquet file in MinIO and the workflow history only keeps a lightweight reference.
|
||||||
|
|
||||||
Notes:
|
Notes:
|
||||||
- `SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES` supports:
|
- `SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES` supports:
|
||||||
- Integer bytes (e.g. `"1572864"`)
|
- Integer bytes (e.g. `"1572864"`)
|
||||||
- Float MiB (e.g. `"1.5"`), converted to bytes as `MiB * 1024 * 1024`
|
- Float MiB (e.g. `"1.5"`), converted to bytes as `MiB * 1024 * 1024`
|
||||||
- Fallback behavior uses `1.5 MiB` when the env var is missing or invalid.
|
- Fallback behavior uses `1.5 MiB` when the env var is missing or invalid.
|
||||||
@@ -751,7 +864,7 @@ After evaluation, the payload is serialized for Temporal as a flat dict:
|
|||||||
- `data` is omitted / set to `None`.
|
- `data` is omitted / set to `None`.
|
||||||
|
|
||||||
When an activity needs pandas operations, it resolves references using:
|
When an activity needs pandas operations, it resolves references using:
|
||||||
- `MinioDataFramePayload.dataframe_from_wire(...)`
|
- `MinioDataFramePayload.retrieve(minio_repo)` — downloads from MinIO or returns inline data as a DataFrame
|
||||||
|
|
||||||
#### MinIO Object Naming (Retention Parsing)
|
#### MinIO Object Naming (Retention Parsing)
|
||||||
|
|
||||||
@@ -952,27 +1065,35 @@ This is the configuration created by the Orchestrator in Temporal.
|
|||||||
```
|
```
|
||||||
laborious/
|
laborious/
|
||||||
├── activities/ # Temporal activity implementations
|
├── activities/ # Temporal activity implementations
|
||||||
│ ├── activities.py # Main activities orchestrator
|
│ ├── activities.py # Main activities aggregator
|
||||||
│ ├── gates.py # Data quality gates and filtering
|
│ ├── gates.py # Data quality gates and filtering
|
||||||
│ ├── mlflow.py # MLFlow model operations
|
│ ├── mlflow.py # MLFlow model operations
|
||||||
|
│ ├── storage.py # PostgreSQL queries and MinIO offload
|
||||||
|
│ ├── model_metrics.py # Drift and regression metrics
|
||||||
│ ├── opc.py # OPC server operations
|
│ ├── opc.py # OPC server operations
|
||||||
│ └── api.py # PI Web API operations
|
│ └── api.py # PI Web API operations
|
||||||
├── workflows/ # Temporal workflow definitions
|
├── workflows/ # Temporal workflow definitions
|
||||||
│ ├── predictions_batch.py # Main batch prediction workflow
|
│ ├── predictions_batch.py # Main batch prediction workflow
|
||||||
│ ├── minimal_retrain.py # Model retraining workflow
|
│ ├── minimal_retrain.py # Model retraining workflow
|
||||||
|
│ ├── drift.py # Data drift detection workflow
|
||||||
|
│ ├── simple_metrics.py # Regression metrics workflow
|
||||||
│ └── sub_workflows/ # Sub-workflow implementations
|
│ └── sub_workflows/ # Sub-workflow implementations
|
||||||
│ ├── prediction_process.py # Core prediction workflow
|
│ ├── prediction_process.py # Core prediction workflow
|
||||||
│ └── format_and_export_prediction.py # Export workflow
|
│ └── format_and_export_prediction.py # Export workflow
|
||||||
├── worker/ # Worker implementation
|
├── worker/ # Worker implementation
|
||||||
│ └── worker.py # Main worker orchestrator
|
│ ├── worker.py # Main worker orchestrator
|
||||||
|
│ └── prepare_worker.py # Worker factory with autoscaling config
|
||||||
├── utils/ # Utility functions
|
├── utils/ # Utility functions
|
||||||
│ ├── connectors_config.py # Database configuration
|
│ ├── connectors_config.py # Environment-driven config builders
|
||||||
|
│ ├── models/ # Data models
|
||||||
|
│ │ └── minio_dataframe_payload.py # MinIO-offloaded DataFrame payload
|
||||||
│ ├── filters/ # Data quality filters
|
│ ├── filters/ # Data quality filters
|
||||||
│ │ ├── conditional_filters.py # Conditional data filters
|
│ │ ├── conditional_filters.py # Conditional data filters
|
||||||
│ │ └── mlflow_filters.py # MLFlow response filters
|
│ │ └── mlflow_filters.py # MLFlow response filters
|
||||||
│ └── repository/ # Data access layer
|
│ └── repository/ # Data access layer
|
||||||
│ ├── model_repository.py # MLFlow model operations
|
│ ├── model_repository.py # MLFlow model operations
|
||||||
│ └── opc_repository.py # OPC server operations
|
│ ├── opc_repository.py # OPC server operations
|
||||||
|
│ └── minio_manager.py # MinIO object storage operations
|
||||||
├── metrics.py # Prometheus metrics definitions
|
├── metrics.py # Prometheus metrics definitions
|
||||||
└── __init__.py
|
└── __init__.py
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ _OBJECT_TIMESTAMP_PATTERN = re.compile(
|
|||||||
)
|
)
|
||||||
|
|
||||||
OFFLOAD_THRESHOLD_BYTES = int(
|
OFFLOAD_THRESHOLD_BYTES = int(
|
||||||
float(getenv('SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES', '1.5')) * 1024 * 1024
|
float(getenv('SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES', '1.5')) * 1024 * 1024
|
||||||
)
|
)
|
||||||
|
|
||||||
# Relative prefix used for storing offloaded training datasets in MinIO.
|
# Relative prefix used for storing offloaded training datasets in MinIO.
|
||||||
|
|||||||
@@ -2,9 +2,9 @@ import os
|
|||||||
import sys
|
import sys
|
||||||
from unittest.mock import MagicMock
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
# The production code converts SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES to int at import-time.
|
# The production code converts SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES to int at import-time.
|
||||||
# Tests must set it to a valid integer string to avoid import errors.
|
# Tests must set it to a valid integer string to avoid import errors.
|
||||||
os.environ.setdefault('SIENTIA_MINIO_OFFLOAD_THRESHOLD_BYTES', '1')
|
os.environ.setdefault('SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES', '1')
|
||||||
|
|
||||||
|
|
||||||
class DummyMinioDataFramePayload:
|
class DummyMinioDataFramePayload:
|
||||||
|
|||||||
@@ -231,6 +231,10 @@ env:
|
|||||||
value: "sa-east-1"
|
value: "sa-east-1"
|
||||||
- name: MINIO_DEFAULT_BUCKET
|
- name: MINIO_DEFAULT_BUCKET
|
||||||
value: "sientia"
|
value: "sientia"
|
||||||
|
- name: MINIO_RETENTION_HOURS
|
||||||
|
value: "24"
|
||||||
|
- name: SIENTIA_MINIO_OFFLOAD_THRESHOLD_MEGABYTES
|
||||||
|
value: "1.5"
|
||||||
|
|
||||||
- name: PI_WEB_API_BASE_URL
|
- name: PI_WEB_API_BASE_URL
|
||||||
value: "https://pivision.votorantimcimentos.com/piwebapi"
|
value: "https://pivision.votorantimcimentos.com/piwebapi"
|
||||||
|
|||||||
Reference in New Issue
Block a user