gitea_admin/sientia-dataops-laborious_temporal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
995ba7900a6648e6c25cacc02568c45add33b75a
sientia-dataops-laborious_t…/README.md
vitor-aignosi 995ba7900a SIENTIAPDE-1182 
Remove Docker configuration files and refactor project structure

- Deleted docker-compose.yml and Dockerfile as part of the project restructuring.
- Updated README.md to reflect changes in project setup and configuration.
- Introduced a new __init__.py file in the laborious package to provide an overview of the system.
- Enhanced documentation across various modules, including metrics, activities, and workflows, to improve clarity and usability.
- Added comprehensive docstrings and comments to key classes and methods for better maintainability.
2025-08-29 13:22:48 -03:00

764 lines
34 KiB
Markdown
Raw Blame History

# Sientia DataOps Laborious
A high-performance, scalable machine learning prediction system built on Temporal.io for industrial data processing and ML model inference. The Laborious system provides enterprise-grade ML model management, batch prediction processing, and real-time data export capabilities with comprehensive data quality validation and monitoring.
## Features
### Core Functionality
- **Batch Prediction Processing**: High-throughput ML model inference using MLFlow models
- **Temporal Workflow Orchestration**: Robust workflow management with automatic retry policies and fault tolerance
- **Data Quality Gates**: Configurable filtering for data validation, MLFlow API responses, and custom validation rules
- **Multi-Model Support**: Flexible ML model management with retention policies and versioning
- **Real-time Data Export**: PostgreSQL persistence and OPC server integration for industrial systems
- **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility
### Advanced Capabilities
- **Incremental Data Processing**: Timestamp-based data loading to avoid reprocessing
- **Configurable Data Retention**: Model retention policies with automatic cleanup
- **Notification System**: Integrated alerting and notification management via MongoDB
- **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support
- **Model Retraining**: Automated model retraining workflows with production model updates
## Architecture
The Laborious system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
### System Overview
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ Temporal Cluster │
│ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │
│ │ Main Worker │ │ Temporal Client │ │ Task Queues │ │
│ │ │◄──►│ │◄──►│ │ │
│ │ - Metrics Server│ │ - Namespace Mgmt │ │ - predictions_batch-queue│ │
│ │ - Notifications │ │ - Runtime Config │ │ - minimal_retrain-queue │ │
│ │ - Lifecycle │ │ - Connection │ │ - Auto-scaling │ │
│ │ - Health Checks │ │ - Security │ │ - Load Balancing │ │
│ └─────────────────┘ └──────────────────┘ └─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Workflow Layer │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ PredictionsBatch│ │ Sub-Workflows │ │
│ │ │ │ │ │
│ │ - Data Loading │ │ - PredictionProcess │ │
│ │ - Configuration │ │ - FormatAndExportPrediction │ │
│ │ - Delegation │ │ - Error Handling │ │
│ └─────────────────┘ └─────────────────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ MinimalRetrain │ │ Model Management │ │
│ │ │ │ │ │
│ │ - Retraining │ │ - Version Control │ │
│ │ - Validation │ │ - Production Updates │ │
│ │ - Deployment │ │ - Quality Assurance │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Activity Layer │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ Data Quality │ │ MLFlow Operations │ │
│ │ │ │ │ │
│ │ - Input Gates │ │ - Model Transform │ │
│ │ - Validation │ │ - Model Prediction │ │
│ │ - Filtering │ │ - Response Validation │ │
│ │ - Policy Mgmt │ │ - Error Handling │ │
│ └─────────────────┘ └─────────────────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ Storage Ops │ │ OPC Operations │ │
│ │ │ │ │ │
│ │ - PostgreSQL │ │ - Server Connections │ │
│ │ - Data Export │ │ - Tag Writing │ │
│ │ - Metrics │ │ - Real-time Export │ │
│ │ - Cleanup │ │ - Error Recovery │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Data Services │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ PostgreSQL │ │ MongoDB │ │
│ │ │ │ │ │
│ │ - Predictions │ │ - Notifications │ │
│ │ - Metadata │ │ - Audit Logs │ │
│ │ - Metrics │ │ - Configuration │ │
│ │ - Cleanup │ │ - User Management │ │
│ └─────────────────┘ └─────────────────────────────┘ │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ MLFlow API │ │ OPC Servers │ │
│ │ │ │ │ │
│ │ - Model Serving │ │ - Real-time Data │ │
│ │ - Transform │ │ - Industrial Integration │ │
│ │ - Prediction │ │ - Security & Auth │ │
│ │ - Versioning │ │ - Load Balancing │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ External Systems │
│ ┌─────────────────┐ ┌─────────────────────────────┐ │
│ │ Prometheus │ │ Kubernetes │ │
│ │ │ │ │ │
│ │ - Metrics │ │ - Orchestration │ │
│ │ - Alerting │ │ - Scaling │ │
│ │ - Dashboards │ │ - Health Checks │ │
│ │ - Monitoring │ │ - Resource Management │ │
│ └─────────────────┘ └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
### Architecture Principles
#### 1. **Separation of Concerns**
- **Worker Layer**: Manages Temporal workers, task queues, and application lifecycle
- **Workflow Layer**: Orchestrates business logic and process coordination
- **Activity Layer**: Implements specific operations and external system interactions
- **Data Layer**: Handles data persistence, caching, and external service connections
#### 2. **Fault Tolerance & Resilience**
- **Automatic Retry Policies**: Configurable retry strategies for transient failures
- **Circuit Breaker Pattern**: Prevents cascading failures in external service calls
- **Graceful Degradation**: System continues operating with reduced functionality
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
#### 3. **Scalability & Performance**
- **Horizontal Scaling**: Multiple worker instances for load distribution
- **Task Queue Isolation**: Separate queues for different workflow types
- **Connection Pooling**: Optimized database and external service connections
- **Asynchronous Processing**: Non-blocking operations for improved throughput
#### 4. **Observability & Monitoring**
- **Prometheus Metrics**: Comprehensive system and business metrics
- **Structured Logging**: Consistent log format with correlation IDs
- **Health Checks**: Endpoint health monitoring and alerting
- **Performance Tracing**: Request flow tracking and bottleneck identification
### Key Components
#### **Worker (`laborious/worker/worker.py`)**
- **Purpose**: Main application orchestrator managing Temporal workers and task queues
- **Responsibilities**:
- Temporal client initialization and connection management
- Worker lifecycle management and graceful shutdown
- Task queue configuration and load balancing
- Prometheus metrics server initialization
- Notification handler setup and configuration
- OPC server connection management
- **Key Features**:
- Automatic scaling with `PollerBehaviorAutoscaling`
- Health check endpoints for Kubernetes liveness/readiness probes
- Graceful shutdown with cleanup procedures
- Multi-instance deployment support
#### **Workflows (`laborious/workflows/`)**
- **PredictionsBatch**: Main entry point for batch prediction pipelines
- **PredictionProcess**: Core prediction pipeline with MLFlow integration
- **FormatAndExportPrediction**: Data formatting and export operations
- **MinimalRetrain**: Automated model retraining and deployment
- **Key Features**:
- Temporal workflow definitions with retry policies
- Child workflow orchestration and delegation
- Comprehensive error handling and recovery
- Configurable timeout and retry strategies
#### **Activities (`laborious/activities/`)**
- **Gates**: Data quality validation and filtering mechanisms
- **MLFlow**: Model transformation and prediction operations
- **OPC**: Real-time data export to industrial OPC servers
- **Activities**: Main activity orchestrator and coordination
- **Key Features**:
- Configurable filter policies and validation rules
- MLFlow model serving integration
- OPC UA client with certificate-based authentication
- Comprehensive error handling and notification
#### **Data Services (`laborious/utils/`)**
- **Connectors**: Database and external service configuration management
- **Repository**: Data access layer for MLFlow and OPC operations
- **Filters**: Data quality validation and MLFlow response filtering
- **Key Features**:
- Environment variable-based configuration
- Connection pool management and optimization
- Security credential management
- Configuration validation and error handling
### Data Flow Architecture
#### **1. Batch Prediction Pipeline**
```
Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform →
MLFlow Prediction → Response Validation → Export (PostgreSQL + OPC)
```
#### **2. Model Retraining Pipeline**
```
Training Data → Model Retraining → Quality Validation →
Production Update → Notification & Monitoring
```
#### **3. Real-time Export Pipeline**
```
Prediction Results → Data Formatting → OPC Server Write →
Success/Failure Metrics → Notification System
```
### Security Architecture
#### **Authentication & Authorization**
- **Certificate-based OPC Authentication**: Secure industrial communication
- **MLFlow API Authentication**: Username/password with secure transmission
- **Database Connection Security**: Encrypted connections with credential management
- **Kubernetes Secrets Integration**: Secure credential storage and access
#### **Network Security**
- **TLS/SSL Encryption**: Secure communication channels
- **Network Isolation**: Kubernetes network policies and service mesh
- **Firewall Rules**: Controlled access to external services
- **VPN Integration**: Secure remote access and management
#### **Data Security**
- **Data Encryption**: At-rest and in-transit encryption
- **Access Control**: Role-based access control (RBAC)
- **Audit Logging**: Comprehensive access and operation logging
- **Data Retention**: Configurable data lifecycle management
## 🔄 Workflows
### 1. Predictions Batch Workflow (`predictions_batch.py`)
The **PredictionsBatch** workflow is the main entry point for batch prediction pipelines. It orchestrates the complete prediction process and implements a robust data loading and processing pattern.
#### Purpose
- **Batch Prediction Orchestration**: Coordinates data loading and prediction processing
- **Data Preparation**: Loads data using custom SQL queries with configurable schemas
- **Workflow Delegation**: Delegates actual prediction processing to the PredictionProcess workflow
- **Configuration Management**: Handles model configuration, filters, and retention policies
#### Execution Flow
1. **Data Loading**: Executes custom SQL query to load data from PostgreSQL
2. **Input Preparation**: Prepares prediction input with metadata and configuration
3. **Workflow Delegation**: Spawns PredictionProcess child workflow for actual processing
4. **Error Handling**: Implements comprehensive error handling with retry policies
#### Key Features
- **Custom Query Support**: Flexible SQL-based data loading
- **Schema Configuration**: Configurable data schema definitions
- **Automatic Retry**: Implements Temporal retry policies for fault tolerance
- **Timeout Management**: 60-second timeout for all activities
- **Comprehensive Error Handling**: Detailed error reporting and notification integration
#### Input Parameters
```json
{
"schedule_name": "hourly_predictions",
"model_name": "temperature_prediction_model",
"model_id": "temp_pred_001",
"query": "SELECT * FROM sensor_data WHERE timestamp > NOW() - INTERVAL '1 hour'",
"schema": {
"timestamp": "datetime",
"temperature": "float",
"humidity": "float"
},
"table_name": "predictions",
"input_filters": {
"EMPTY_DATA": {"POLICY": "STOP"}
},
"mlflow_transform_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"mlflow_predict_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"model_retention": 60,
"path_priority": ["STOP", "CONTINUE", "REPEAT"],
"opc_output_config": {
"server_id": "opc_server_1",
"tags": ["prediction_output"]
}
}
```
### 2. Prediction Process Workflow (`prediction_process.py`)
The **PredictionProcess** workflow implements the core prediction pipeline for ML model inference. It handles data quality validation, MLFlow model interactions, and prediction processing.
#### Purpose
- **Data Quality Validation**: Applies configurable filters for data integrity
- **MLFlow Integration**: Manages model transformation and prediction requests
- **Response Validation**: Filters MLFlow API responses for quality assurance
- **Prediction Export**: Delegates prediction formatting and export operations
#### Execution Flow
1. **Timestamp Retrieval**: Gets the last processed timestamp for incremental processing
2. **Input Data Gate**: Applies configured filters for data quality validation
3. **Path Decision**: Determines processing path based on filter results
4. **MLFlow Transform**: Requests data transformation using MLFlow models
5. **Response Validation**: Filters transform responses for quality assurance
6. **MLFlow Prediction**: Executes prediction using transformed data
7. **Content Validation**: Filters prediction responses for final quality check
8. **Export Delegation**: Delegates to FormatAndExportPrediction workflow
#### Key Features
- **Configurable Quality Gates**: Multiple filter types with policy-based configuration
- **Flexible Path Handling**: Configurable decision paths (STOP, CONTINUE, REPEAT)
- **MLFlow Integration**: Comprehensive model management and inference
- **Incremental Processing**: Timestamp-based data processing optimization
- **Comprehensive Monitoring**: Detailed metrics and error reporting
#### Input Parameters
```json
{
"metadata": {
"schedule_name": "hourly_predictions",
"model_name": "temperature_prediction_model",
"model_id": "temp_pred_001",
"workflow_name": "predictions_batch"
},
"data": {...},
"schema": {...},
"table_name": "predictions",
"model_id": "temp_pred_001",
"model_name": "temperature_prediction_model",
"input_filters": {
"EMPTY_DATA": {"POLICY": "STOP"},
"SPECIFIC_VARIABLES_NULL_VALUES": {
"POLICY": "STOP",
"config": {"variables": ["temperature", "humidity"]}
}
},
"mlflow_transform_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"mlflow_predict_filters": {
"API_ERROR": {"POLICY": "STOP"},
"NAN_VALUES": {"POLICY": "STOP"}
},
"model_retention": 60,
"path_priority": ["STOP", "CONTINUE", "REPEAT"],
"opc_output_config": {...}
}
```
### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`)
The **FormatAndExportPrediction** workflow handles prediction data formatting and export operations to multiple destinations.
#### Purpose
- **Data Formatting**: Formats prediction data for different output destinations
- **PostgreSQL Export**: Persists predictions to database with metrics
- **OPC Integration**: Writes predictions to OPC servers for real-time access
- **Metrics Recording**: Tracks export operations and performance metrics
#### Execution Flow
1. **Path Decision**: Determines formatting path based on configuration
2. **Data Formatting**: Formats prediction data for specific output requirements
3. **PostgreSQL Export**: Writes formatted predictions to database
4. **OPC Export**: Writes predictions to OPC servers
5. **Metrics Recording**: Records export performance and success metrics
#### Key Features
- **Flexible Formatting**: Configurable output formats for different destinations
- **Multi-Destination Export**: PostgreSQL and OPC server integration
- **Performance Monitoring**: Comprehensive metrics for export operations
- **Error Handling**: Robust error handling with notification integration
### 4. Minimal Retrain Workflow (`minimal_retrain.py`)
The **MinimalRetrain** workflow handles automated model retraining and production model updates.
#### Purpose
- **Model Retraining**: Automates ML model retraining processes
- **Production Updates**: Manages production model version updates
- **Data Export**: Exports training data for model development
- **Quality Assurance**: Ensures model quality before production deployment
#### Execution Flow
1. **Data Loading**: Loads training data using custom queries
2. **Model Retraining**: Executes model retraining process
3. **Quality Validation**: Validates retrained model performance
4. **Production Update**: Updates production model if quality criteria met
5. **Data Export**: Exports training data for analysis
## 📋 Prerequisites
- Python 3.11+
- Temporal server/cluster
- PostgreSQL database
- MLFlow server
- OPC server(s)
- MongoDB server (for notifications)
**Note**: External dependencies must be available either through:
- Kubernetes cluster deployment
- Docker Compose setup
- Cloud-managed services
- Local installations
## 🚀 Installation
### Local Development Setup
1. **Clone the repository**
```bash
git clone <repository-url>
cd sientia-dataops-laborious
```
2. **Create virtual environment**
```bash
python3.11 -m venv venv
source ./venv/bin/activate
```
3. **Install dependencies**
```bash
pip install -r requirements.txt
```
4. **Create environment configuration file**
```bash
cp .env.example .env
# Edit .env with your connection details
```
5. **Configure external dependencies**
You'll need to set up port forwarding or connections to external services. For example:
```bash
# Port forwarding from Kubernetes cluster
kubectl port-forward svc/postgresql 5432:5432
kubectl port-forward svc/mlflow 5000:5000
kubectl port-forward svc/mongodb 27017:27017
# Or connect to external services
# Ensure services are accessible on localhost with appropriate ports
```
## 📦 How to Run
### Running the Laborious Application
Use the provided script to run the application locally:
```bash
# Make script executable (first time only)
chmod +x run_local.sh
# Run the application
./run_local.sh
```
The script will:
- Activate the virtual environment
- Load environment variables from `.env`
- Start the laborious worker application
### Running Tests and Coverage
Use the provided script to run tests with coverage:
```bash
# Make script executable (first time only)
chmod +x run_coverage.sh
# Run tests with coverage
./run_coverage.sh
```
The script will:
- Activate the virtual environment
- Run pytest with coverage reporting
- Generate HTML coverage report
- Open the coverage report in your browser
### Manual Test Execution
You can also run tests manually:
```bash
# Activate virtual environment
source ./venv/bin/activate
# Run all tests
pytest
# Run with coverage
pytest --cov=laborious --cov-report=html
# Run specific test categories
pytest tests/activities/
pytest tests/workflow/
```
### Manual Application Execution
For manual execution without scripts:
```bash
# Activate virtual environment
source ./venv/bin/activate
# Load environment variables (if using .env file)
if [ -f .env ]; then
export $(cat .env | grep -v '^#' | xargs)
fi
# Start the laborious worker
python -m laborious.worker.worker
```
## 🧪 Testing
### Test Structure
```
tests/
├── activities/ # Activity implementation tests
├── workflow/ # Workflow orchestration tests
├── utils/ # Utility function tests
└── integration/ # End-to-end workflow tests
```
### Test Execution
```bash
# Install test dependencies
pip install pytest pytest-cov pytest-asyncio
# Run tests with coverage
pytest --cov=laborious --cov-report=html
# Run specific test modules
pytest tests/activities/test_gates.py
pytest tests/workflow/test_predictions_batch.py
```
## 📊 Monitoring and Metrics
The Laborious system exposes comprehensive Prometheus metrics:
### Application Metrics
- `app_up`: Application health status (1=healthy, 0=unhealthy)
- `laborious_predictions_written_count`: Prediction export operation count
- `laborious_prediction_confidence_monitor`: Prediction confidence monitoring
- `laborious_prediction_response_time_monitor`: Prediction response time monitoring
### MLFlow Metrics
- Model transformation and prediction success rates
- API response times and error rates
- Model retention and versioning metrics
### Export Metrics
- PostgreSQL export operation counts and response times
- OPC server write operations and performance
- Data quality filter pass/fail rates
## ⚙️ Configuration
### Environment Variables
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
| `TEMPORAL_NAMESPACE` | Temporal namespace | `laborious` | No |
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
| `POSTGRES_PASSWORD` | PostgreSQL password | `sientia` | Yes |
| `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
| `MLFLOW_HOST` | MLFlow server hostname | `localhost` | Yes |
| `MLFLOW_PORT` | MLFlow server port | `5000` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `admin` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `admin` | Yes |
| `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
| `MONGODB_URL` | MongoDB connection URI | `localhost:27017` | Yes |
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
### OPC Configuration
For multiple OPC servers, use the `OPC_CONFIG` environment variable:
```json
{
"opc_server_1": {
"url": "opc.tcp://server1:4840",
"name": "Server1",
"server_uri": "urn:server1:opcua",
"cert_path": "/path/to/cert.pem",
"private_key_path": "/path/to/key.pem",
"server_cert_path": "/path/to/server_cert.pem",
"reconnection_interval": 5000
},
"opc_server_2": {
"url": "opc.tcp://server2:4840",
"name": "Server2",
"server_uri": "urn:server2:opcua",
"cert_path": "/path/to/cert.pem",
"private_key_path": "/path/to/key.pem",
"server_cert_path": "/path/to/server_cert.pem",
"reconnection_interval": 5000
}
}
```
For single OPC server, use individual environment variables:
- `OPC_URL`
- `OPC_NAME`
- `OPC_SERVER_URI`
- `OPC_CERT_PATH`
- `OPC_PRIVATE_KEY_PATH`
- `OPC_SERVER_CERT_PATH`
- `OPC_RECONNECTION_INTERVAL`
### Workflow Configuration
Workflows are configured through input parameters and filter policies:
```json
{
"input_filters": {
"EMPTY_DATA": {"POLICY": "STOP"},
"SPECIFIC_VARIABLES_NULL_VALUES": {
"POLICY": "STOP",
"config": {"variables": ["temperature", "humidity"]}
}
},
"mlflow_transform_filters": {
"API_ERROR": {"POLICY": "STOP"}
},
"mlflow_predict_filters": {
"API_ERROR": {"POLICY": "STOP"},
"NAN_VALUES": {"POLICY": "STOP"}
},
"path_priority": ["STOP", "CONTINUE", "REPEAT"],
"model_retention": 60
}
```
## 🔧 Development
### Project Structure
```
laborious/
├── activities/ # Temporal activity implementations
│ ├── activities.py # Main activities orchestrator
│ ├── gates.py # Data quality gates and filtering
│ ├── mlflow.py # MLFlow model operations
│ └── opc.py # OPC server operations
├── workflow/ # Temporal workflow definitions
│ ├── predictions_batch.py # Main batch prediction workflow
│ ├── minimal_retrain.py # Model retraining workflow
│ └── sub_workflows/ # Sub-workflow implementations
│ ├── prediction_process.py # Core prediction workflow
│ └── format_and_export_prediction.py # Export workflow
├── worker/ # Worker implementation
│ └── worker.py # Main worker orchestrator
├── utils/ # Utility functions
│ ├── connectors_config.py # Database configuration
│ ├── filters/ # Data quality filters
│ │ ├── conditional_filters.py # Conditional data filters
│ │ └── mlflow_filters.py # MLFlow response filters
│ └── repository/ # Data access layer
│ ├── model_repository.py # MLFlow model operations
│ └── opc_repository.py # OPC server operations
├── metrics.py # Prometheus metrics definitions
└── __init__.py
```
### Adding New Features
1. **Follow Temporal patterns** for new workflows and activities
2. **Add comprehensive docstrings** for all public methods
3. **Include Prometheus metrics** for monitoring
4. **Add unit tests** for new functionality
5. **Update this README** with new features and configuration
## 🐛 Troubleshooting
### Common Issues
1. **Temporal Connection Failures**
- Verify Temporal server is running and accessible
- Check namespace configuration and permissions
- Review server logs for connection issues
2. **MLFlow Connection Issues**
- Verify MLFlow server is running and accessible
- Check authentication credentials and permissions
- Ensure model names and versions exist
3. **Database Connection Issues**
- Verify PostgreSQL service is running
- Check connection credentials and network access
- Ensure proper connection pool configuration
4. **OPC Connection Failures**
- Verify OPC server is accessible
- Check certificate and key file paths
- Review OPC server logs for connection issues
5. **Workflow Execution Failures**
- Review activity error logs and notifications
- Check data quality filter configurations
- Verify input data format and required fields
### Debug Mode
Enable debug logging by setting the log level:
```bash
export LOG_LEVEL=DEBUG
```
## ⚡ Performance Tuning
### Key Parameters
- **Worker Concurrency**: Adjust `max_concurrent_workflow_tasks` and `max_concurrent_activities`
- **Connection Pools**: Optimize database connection pool sizes
- **Model Retention**: Configure MLFlow model retention based on requirements
- **Batch Sizes**: Adjust data processing batch sizes for optimal throughput
### Scaling Considerations
- **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Use multiple task queues for different workflow types
- **Database Performance**: Optimize indexes and connection pooling
- **MLFlow Performance**: Configure appropriate model serving resources
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes with comprehensive testing
4. Update documentation and docstrings
5. Submit a pull request
### Code Quality Standards
- Follow PEP 8 style guidelines
- Include comprehensive docstrings for all public methods
- Maintain test coverage above 80%
- Use type hints where appropriate
- Follow Temporal.io best practices
## 📄 License
This project is licensed under the terms specified in the LICENSE file.
## 🆘 Support
For support and questions:
- Check the troubleshooting section above
- Review the metrics and logs for error patterns
- Open an issue in the project repository
- Contact the development team
---
**Note**: The Laborious system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.
Reference in New Issue View Git Blame Copy Permalink