# Sientia DataOps Laborious A high-performance, scalable machine learning prediction system built on Temporal.io for industrial data processing and ML model inference. The Laborious system provides enterprise-grade ML model management, batch prediction processing, and real-time data export capabilities with comprehensive data quality validation and monitoring. ## Features ### Core Functionality - **Batch Prediction Processing**: High-throughput ML model inference using MLFlow models - **Temporal Workflow Orchestration**: Robust workflow management with automatic retry policies and fault tolerance - **Data Quality Gates**: Configurable filtering for data validation, MLFlow API responses, and custom validation rules - **Multi-Model Support**: Flexible ML model management with retention policies and versioning - **Real-time Data Export**: PostgreSQL persistence and OPC server integration for industrial systems - **Comprehensive Monitoring**: Prometheus metrics and detailed logging for operational visibility ### Advanced Capabilities - **Incremental Data Processing**: Timestamp-based data loading to avoid reprocessing - **Configurable Data Retention**: Model retention policies with automatic cleanup - **Notification System**: Integrated alerting and notification management via MongoDB - **Scalable Architecture**: Kubernetes-ready deployment with horizontal scaling support - **Model Retraining**: Automated model retraining workflows with production model updates ## Architecture The Laborious system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments. ### System Overview ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ Temporal Cluster │ │ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │ │ │ Main Worker │ │ Temporal Client │ │ Task Queues │ │ │ │ │◄──►│ │◄──►│ │ │ │ │ - Metrics Server│ │ - Namespace Mgmt │ │ - predictions_batch-queue│ │ │ │ - Notifications │ │ - Runtime Config │ │ - minimal_retrain-queue │ │ │ │ - Lifecycle │ │ - Connection │ │ - Auto-scaling │ │ │ │ - Health Checks │ │ - Security │ │ - Load Balancing │ │ │ └─────────────────┘ └──────────────────┘ └─────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Workflow Layer │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ PredictionsBatch│ │ Sub-Workflows │ │ │ │ │ │ │ │ │ │ - Data Loading │ │ - PredictionProcess │ │ │ │ - Configuration │ │ - FormatAndExportPrediction │ │ │ │ - Delegation │ │ - Error Handling │ │ │ └─────────────────┘ └─────────────────────────────┘ │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ MinimalRetrain │ │ Model Management │ │ │ │ │ │ │ │ │ │ - Retraining │ │ - Version Control │ │ │ │ - Validation │ │ - Production Updates │ │ │ │ - Deployment │ │ - Quality Assurance │ │ │ └─────────────────┘ └─────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Activity Layer │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ Data Quality │ │ MLFlow Operations │ │ │ │ │ │ │ │ │ │ - Input Gates │ │ - Model Transform │ │ │ │ - Validation │ │ - Model Prediction │ │ │ │ - Filtering │ │ - Response Validation │ │ │ │ - Policy Mgmt │ │ - Error Handling │ │ │ └─────────────────┘ └─────────────────────────────┘ │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ Storage Ops │ │ OPC Operations │ │ │ │ │ │ │ │ │ │ - PostgreSQL │ │ - Server Connections │ │ │ │ - Data Export │ │ - Tag Writing │ │ │ │ - Metrics │ │ - Real-time Export │ │ │ │ - Cleanup │ │ - Error Recovery │ │ │ └─────────────────┘ └─────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Data Services │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ PostgreSQL │ │ MongoDB │ │ │ │ │ │ │ │ │ │ - Predictions │ │ - Notifications │ │ │ │ - Metadata │ │ - Audit Logs │ │ │ │ - Metrics │ │ - Configuration │ │ │ │ - Cleanup │ │ - User Management │ │ │ └─────────────────┘ └─────────────────────────────┘ │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ MLFlow API │ │ OPC Servers │ │ │ │ │ │ │ │ │ │ - Model Serving │ │ - Real-time Data │ │ │ │ - Transform │ │ - Industrial Integration │ │ │ │ - Prediction │ │ - Security & Auth │ │ │ │ - Versioning │ │ - Load Balancing │ │ │ └─────────────────┘ └─────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ External Systems │ │ ┌─────────────────┐ ┌─────────────────────────────┐ │ │ │ Prometheus │ │ Kubernetes │ │ │ │ │ │ │ │ │ │ - Metrics │ │ - Orchestration │ │ │ │ - Alerting │ │ - Scaling │ │ │ │ - Dashboards │ │ - Health Checks │ │ │ │ - Monitoring │ │ - Resource Management │ │ │ └─────────────────┘ └─────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` ### Architecture Principles #### 1. **Separation of Concerns** - **Worker Layer**: Manages Temporal workers, task queues, and application lifecycle - **Workflow Layer**: Orchestrates business logic and process coordination - **Activity Layer**: Implements specific operations and external system interactions - **Data Layer**: Handles data persistence, caching, and external service connections #### 2. **Fault Tolerance & Resilience** - **Automatic Retry Policies**: Configurable retry strategies for transient failures - **Circuit Breaker Pattern**: Prevents cascading failures in external service calls - **Graceful Degradation**: System continues operating with reduced functionality - **Comprehensive Error Handling**: Detailed error reporting and notification integration #### 3. **Scalability & Performance** - **Horizontal Scaling**: Multiple worker instances for load distribution - **Task Queue Isolation**: Separate queues for different workflow types - **Connection Pooling**: Optimized database and external service connections - **Asynchronous Processing**: Non-blocking operations for improved throughput #### 4. **Observability & Monitoring** - **Prometheus Metrics**: Comprehensive system and business metrics - **Structured Logging**: Consistent log format with correlation IDs - **Health Checks**: Endpoint health monitoring and alerting - **Performance Tracing**: Request flow tracking and bottleneck identification ### Key Components #### **Worker (`laborious/worker/worker.py`)** - **Purpose**: Main application orchestrator managing Temporal workers and task queues - **Responsibilities**: - Temporal client initialization and connection management - Worker lifecycle management and graceful shutdown - Task queue configuration and load balancing - Prometheus metrics server initialization - Notification handler setup and configuration - OPC server connection management - **Key Features**: - Automatic scaling with `PollerBehaviorAutoscaling` - Health check endpoints for Kubernetes liveness/readiness probes - Graceful shutdown with cleanup procedures - Multi-instance deployment support #### **Workflows (`laborious/workflows/`)** - **PredictionsBatch**: Main entry point for batch prediction pipelines - **PredictionProcess**: Core prediction pipeline with MLFlow integration - **FormatAndExportPrediction**: Data formatting and export operations - **MinimalRetrain**: Automated model retraining and deployment - **Key Features**: - Temporal workflow definitions with retry policies - Child workflow orchestration and delegation - Comprehensive error handling and recovery - Configurable timeout and retry strategies #### **Activities (`laborious/activities/`)** - **Gates**: Data quality validation and filtering mechanisms - **MLFlow**: Model transformation and prediction operations - **OPC**: Real-time data export to industrial OPC servers - **Activities**: Main activity orchestrator and coordination - **Key Features**: - Configurable filter policies and validation rules - MLFlow model serving integration - OPC UA client with certificate-based authentication - Comprehensive error handling and notification #### **Data Services (`laborious/utils/`)** - **Connectors**: Database and external service configuration management - **Repository**: Data access layer for MLFlow and OPC operations - **Filters**: Data quality validation and MLFlow response filtering - **Key Features**: - Environment variable-based configuration - Connection pool management and optimization - Security credential management - Configuration validation and error handling ### Data Flow Architecture #### **1. Batch Prediction Pipeline** ``` Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform → MLFlow Prediction → Response Validation → Export (PostgreSQL + OPC) ``` #### **2. Model Retraining Pipeline** ``` Training Data → Model Retraining → Quality Validation → Production Update → Notification & Monitoring ``` #### **3. Real-time Export Pipeline** ``` Prediction Results → Data Formatting → OPC Server Write → Success/Failure Metrics → Notification System ``` ### Security Architecture #### **Authentication & Authorization** - **Certificate-based OPC Authentication**: Secure industrial communication - **MLFlow API Authentication**: Username/password with secure transmission - **Database Connection Security**: Encrypted connections with credential management - **Kubernetes Secrets Integration**: Secure credential storage and access #### **Network Security** - **TLS/SSL Encryption**: Secure communication channels - **Network Isolation**: Kubernetes network policies and service mesh - **Firewall Rules**: Controlled access to external services - **VPN Integration**: Secure remote access and management #### **Data Security** - **Data Encryption**: At-rest and in-transit encryption - **Access Control**: Role-based access control (RBAC) - **Audit Logging**: Comprehensive access and operation logging - **Data Retention**: Configurable data lifecycle management ## 🔄 Workflows ### 1. Predictions Batch Workflow (`predictions_batch.py`) The **PredictionsBatch** workflow is the main entry point for batch prediction pipelines. It orchestrates the complete prediction process and implements a robust data loading and processing pattern. #### Purpose - **Batch Prediction Orchestration**: Coordinates data loading and prediction processing - **Data Preparation**: Loads data using custom SQL queries with configurable schemas - **Workflow Delegation**: Delegates actual prediction processing to the PredictionProcess workflow - **Configuration Management**: Handles model configuration, filters, and retention policies #### Execution Flow 1. **Data Loading**: Executes custom SQL query to load data from PostgreSQL 2. **Input Preparation**: Prepares prediction input with metadata and configuration 3. **Workflow Delegation**: Spawns PredictionProcess child workflow for actual processing 4. **Error Handling**: Implements comprehensive error handling with retry policies #### Key Features - **Custom Query Support**: Flexible SQL-based data loading - **Schema Configuration**: Configurable data schema definitions - **Automatic Retry**: Implements Temporal retry policies for fault tolerance - **Timeout Management**: 60-second timeout for all activities - **Comprehensive Error Handling**: Detailed error reporting and notification integration #### Input Parameters ```json { "schedule_name": "hourly_predictions", "model_name": "temperature_prediction_model", "model_id": "temp_pred_001", "query": "SELECT * FROM sensor_data WHERE timestamp > NOW() - INTERVAL '1 hour'", "schema": { "timestamp": "datetime", "temperature": "float", "humidity": "float" }, "table_name": "predictions", "input_filters": { "EMPTY_DATA": {"POLICY": "STOP"} }, "mlflow_transform_filters": { "API_ERROR": {"POLICY": "STOP"} }, "mlflow_predict_filters": { "API_ERROR": {"POLICY": "STOP"} }, "model_retention": 60, "path_priority": ["STOP", "CONTINUE", "REPEAT"], "opc_output_config": { "server_id": "opc_server_1", "tags": ["prediction_output"] } } ``` ### 2. Prediction Process Workflow (`prediction_process.py`) The **PredictionProcess** workflow implements the core prediction pipeline for ML model inference. It handles data quality validation, MLFlow model interactions, and prediction processing. #### Purpose - **Data Quality Validation**: Applies configurable filters for data integrity - **MLFlow Integration**: Manages model transformation and prediction requests - **Response Validation**: Filters MLFlow API responses for quality assurance - **Prediction Export**: Delegates prediction formatting and export operations #### Execution Flow 1. **Timestamp Retrieval**: Gets the last processed timestamp for incremental processing 2. **Input Data Gate**: Applies configured filters for data quality validation 3. **Path Decision**: Determines processing path based on filter results 4. **MLFlow Transform**: Requests data transformation using MLFlow models 5. **Response Validation**: Filters transform responses for quality assurance 6. **MLFlow Prediction**: Executes prediction using transformed data 7. **Content Validation**: Filters prediction responses for final quality check 8. **Export Delegation**: Delegates to FormatAndExportPrediction workflow #### Key Features - **Configurable Quality Gates**: Multiple filter types with policy-based configuration - **Flexible Path Handling**: Configurable decision paths (STOP, CONTINUE, REPEAT) - **MLFlow Integration**: Comprehensive model management and inference - **Incremental Processing**: Timestamp-based data processing optimization - **Comprehensive Monitoring**: Detailed metrics and error reporting #### Input Parameters ```json { "metadata": { "schedule_name": "hourly_predictions", "model_name": "temperature_prediction_model", "model_id": "temp_pred_001", "workflow_name": "predictions_batch" }, "data": {...}, "schema": {...}, "table_name": "predictions", "model_id": "temp_pred_001", "model_name": "temperature_prediction_model", "input_filters": { "EMPTY_DATA": {"POLICY": "STOP"}, "SPECIFIC_VARIABLES_NULL_VALUES": { "POLICY": "STOP", "config": {"variables": ["temperature", "humidity"]} } }, "mlflow_transform_filters": { "API_ERROR": {"POLICY": "STOP"} }, "mlflow_predict_filters": { "API_ERROR": {"POLICY": "STOP"}, "NAN_VALUES": {"POLICY": "STOP"} }, "model_retention": 60, "path_priority": ["STOP", "CONTINUE", "REPEAT"], "opc_output_config": {...} } ``` ### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`) The **FormatAndExportPrediction** workflow handles prediction data formatting and export operations to multiple destinations. #### Purpose - **Data Formatting**: Formats prediction data for different output destinations - **PostgreSQL Export**: Persists predictions to database with metrics - **OPC Integration**: Writes predictions to OPC servers for real-time access - **Metrics Recording**: Tracks export operations and performance metrics #### Execution Flow 1. **Path Decision**: Determines formatting path based on configuration 2. **Data Formatting**: Formats prediction data for specific output requirements 3. **PostgreSQL Export**: Writes formatted predictions to database 4. **OPC Export**: Writes predictions to OPC servers 5. **Metrics Recording**: Records export performance and success metrics #### Key Features - **Flexible Formatting**: Configurable output formats for different destinations - **Multi-Destination Export**: PostgreSQL and OPC server integration - **Performance Monitoring**: Comprehensive metrics for export operations - **Error Handling**: Robust error handling with notification integration ### 4. Minimal Retrain Workflow (`minimal_retrain.py`) The **MinimalRetrain** workflow handles automated model retraining and production model updates. #### Purpose - **Model Retraining**: Automates ML model retraining processes - **Production Updates**: Manages production model version updates - **Data Export**: Exports training data for model development - **Quality Assurance**: Ensures model quality before production deployment #### Execution Flow 1. **Data Loading**: Loads training data using custom queries 2. **Model Retraining**: Executes model retraining process 3. **Quality Validation**: Validates retrained model performance 4. **Production Update**: Updates production model if quality criteria met 5. **Data Export**: Exports training data for analysis ## 📋 Prerequisites - Python 3.11+ - Temporal server/cluster - PostgreSQL database - MLFlow server - OPC server(s) - MongoDB server (for notifications) **Note**: External dependencies must be available either through: - Kubernetes cluster deployment - Docker Compose setup - Cloud-managed services - Local installations ## 🚀 Installation ### Local Development Setup 1. **Clone the repository** ```bash git clone cd sientia-dataops-laborious ``` 2. **Create virtual environment** ```bash python3.11 -m venv venv source ./venv/bin/activate ``` 3. **Install dependencies** ```bash pip install -r requirements.txt ``` 4. **Create environment configuration file** ```bash cp .env.example .env # Edit .env with your connection details ``` 5. **Configure external dependencies** You'll need to set up port forwarding or connections to external services. For example: ```bash # Port forwarding from Kubernetes cluster kubectl port-forward svc/postgresql 5432:5432 kubectl port-forward svc/mlflow 5000:5000 kubectl port-forward svc/mongodb 27017:27017 # Or connect to external services # Ensure services are accessible on localhost with appropriate ports ``` ## 📦 How to Run ### Running the Laborious Application Use the provided script to run the application locally: ```bash # Make script executable (first time only) chmod +x run_local.sh # Run the application ./run_local.sh ``` The script will: - Activate the virtual environment - Load environment variables from `.env` - Start the laborious worker application ### Running Tests and Coverage Use the provided script to run tests with coverage: ```bash # Make script executable (first time only) chmod +x run_coverage.sh # Run tests with coverage ./run_coverage.sh ``` The script will: - Activate the virtual environment - Run pytest with coverage reporting - Generate HTML coverage report - Open the coverage report in your browser ### Manual Test Execution You can also run tests manually: ```bash # Activate virtual environment source ./venv/bin/activate # Run all tests pytest # Run with coverage pytest --cov=laborious --cov-report=html # Run specific test categories pytest tests/activities/ pytest tests/workflow/ ``` ### Manual Application Execution For manual execution without scripts: ```bash # Activate virtual environment source ./venv/bin/activate # Load environment variables (if using .env file) if [ -f .env ]; then export $(cat .env | grep -v '^#' | xargs) fi # Start the laborious worker python -m laborious.worker.worker ``` ## 🧪 Testing ### Test Structure ``` tests/ ├── activities/ # Activity implementation tests ├── workflow/ # Workflow orchestration tests ├── utils/ # Utility function tests └── integration/ # End-to-end workflow tests ``` ### Test Execution ```bash # Install test dependencies pip install pytest pytest-cov pytest-asyncio # Run tests with coverage pytest --cov=laborious --cov-report=html # Run specific test modules pytest tests/activities/test_gates.py pytest tests/workflow/test_predictions_batch.py ``` ## 📊 Monitoring and Metrics The Laborious system exposes comprehensive Prometheus metrics: ### Application Metrics - `app_up`: Application health status (1=healthy, 0=unhealthy) - `laborious_predictions_written_count`: Prediction export operation count - `laborious_prediction_confidence_monitor`: Prediction confidence monitoring - `laborious_prediction_response_time_monitor`: Prediction response time monitoring ### MLFlow Metrics - Model transformation and prediction success rates - API response times and error rates - Model retention and versioning metrics ### Export Metrics - PostgreSQL export operation counts and response times - OPC server write operations and performance - Data quality filter pass/fail rates ## ⚙️ Configuration ### Environment Variables | Variable | Description | Default | Required | |----------|-------------|---------|----------| | `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes | | `TEMPORAL_NAMESPACE` | Temporal namespace | `laborious` | No | | `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes | | `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes | | `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes | | `POSTGRES_PASSWORD` | PostgreSQL password | `sientia` | Yes | | `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes | | `MLFLOW_HOST` | MLFlow server hostname | `localhost` | Yes | | `MLFLOW_PORT` | MLFlow server port | `5000` | Yes | | `MLFLOW_USERNAME` | MLFlow username | `admin` | Yes | | `MLFLOW_PASSWORD` | MLFlow password | `admin` | Yes | | `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No | | `MONGODB_URL` | MongoDB connection URI | `localhost:27017` | Yes | | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No | | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No | ### OPC Configuration For multiple OPC servers, use the `OPC_CONFIG` environment variable: ```json { "opc_server_1": { "url": "opc.tcp://server1:4840", "name": "Server1", "server_uri": "urn:server1:opcua", "cert_path": "/path/to/cert.pem", "private_key_path": "/path/to/key.pem", "server_cert_path": "/path/to/server_cert.pem", "reconnection_interval": 5000 }, "opc_server_2": { "url": "opc.tcp://server2:4840", "name": "Server2", "server_uri": "urn:server2:opcua", "cert_path": "/path/to/cert.pem", "private_key_path": "/path/to/key.pem", "server_cert_path": "/path/to/server_cert.pem", "reconnection_interval": 5000 } } ``` For single OPC server, use individual environment variables: - `OPC_URL` - `OPC_NAME` - `OPC_SERVER_URI` - `OPC_CERT_PATH` - `OPC_PRIVATE_KEY_PATH` - `OPC_SERVER_CERT_PATH` - `OPC_RECONNECTION_INTERVAL` ### Workflow Configuration Workflows are configured through input parameters and filter policies: ```json { "input_filters": { "EMPTY_DATA": {"POLICY": "STOP"}, "SPECIFIC_VARIABLES_NULL_VALUES": { "POLICY": "STOP", "config": {"variables": ["temperature", "humidity"]} } }, "mlflow_transform_filters": { "API_ERROR": {"POLICY": "STOP"} }, "mlflow_predict_filters": { "API_ERROR": {"POLICY": "STOP"}, "NAN_VALUES": {"POLICY": "STOP"} }, "path_priority": ["STOP", "CONTINUE", "REPEAT"], "model_retention": 60 } ``` ## 🔧 Development ### Project Structure ``` laborious/ ├── activities/ # Temporal activity implementations │ ├── activities.py # Main activities orchestrator │ ├── gates.py # Data quality gates and filtering │ ├── mlflow.py # MLFlow model operations │ └── opc.py # OPC server operations ├── workflow/ # Temporal workflow definitions │ ├── predictions_batch.py # Main batch prediction workflow │ ├── minimal_retrain.py # Model retraining workflow │ └── sub_workflows/ # Sub-workflow implementations │ ├── prediction_process.py # Core prediction workflow │ └── format_and_export_prediction.py # Export workflow ├── worker/ # Worker implementation │ └── worker.py # Main worker orchestrator ├── utils/ # Utility functions │ ├── connectors_config.py # Database configuration │ ├── filters/ # Data quality filters │ │ ├── conditional_filters.py # Conditional data filters │ │ └── mlflow_filters.py # MLFlow response filters │ └── repository/ # Data access layer │ ├── model_repository.py # MLFlow model operations │ └── opc_repository.py # OPC server operations ├── metrics.py # Prometheus metrics definitions └── __init__.py ``` ### Adding New Features 1. **Follow Temporal patterns** for new workflows and activities 2. **Add comprehensive docstrings** for all public methods 3. **Include Prometheus metrics** for monitoring 4. **Add unit tests** for new functionality 5. **Update this README** with new features and configuration ## 🐛 Troubleshooting ### Common Issues 1. **Temporal Connection Failures** - Verify Temporal server is running and accessible - Check namespace configuration and permissions - Review server logs for connection issues 2. **MLFlow Connection Issues** - Verify MLFlow server is running and accessible - Check authentication credentials and permissions - Ensure model names and versions exist 3. **Database Connection Issues** - Verify PostgreSQL service is running - Check connection credentials and network access - Ensure proper connection pool configuration 4. **OPC Connection Failures** - Verify OPC server is accessible - Check certificate and key file paths - Review OPC server logs for connection issues 5. **Workflow Execution Failures** - Review activity error logs and notifications - Check data quality filter configurations - Verify input data format and required fields ### Debug Mode Enable debug logging by setting the log level: ```bash export LOG_LEVEL=DEBUG ``` ## ⚡ Performance Tuning ### Key Parameters - **Worker Concurrency**: Adjust `max_concurrent_workflow_tasks` and `max_concurrent_activities` - **Connection Pools**: Optimize database connection pool sizes - **Model Retention**: Configure MLFlow model retention based on requirements - **Batch Sizes**: Adjust data processing batch sizes for optimal throughput ### Scaling Considerations - **Horizontal Scaling**: Deploy multiple worker instances - **Task Queue Distribution**: Use multiple task queues for different workflow types - **Database Performance**: Optimize indexes and connection pooling - **MLFlow Performance**: Configure appropriate model serving resources ## 🤝 Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes with comprehensive testing 4. Update documentation and docstrings 5. Submit a pull request ### Code Quality Standards - Follow PEP 8 style guidelines - Include comprehensive docstrings for all public methods - Maintain test coverage above 80% - Use type hints where appropriate - Follow Temporal.io best practices ## 📄 License This project is licensed under the terms specified in the LICENSE file. ## 🆘 Support For support and questions: - Check the troubleshooting section above - Review the metrics and logs for error patterns - Open an issue in the project repository - Contact the development team --- **Note**: The Laborious system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.