diff --git a/README.md b/README.md index 918107a..9ace5a8 100644 --- a/README.md +++ b/README.md @@ -23,95 +23,6 @@ A high-performance, scalable machine learning prediction system built on Tempora The Laborious system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments. -### System Overview - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ Temporal Cluster │ -│ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────────────┐ │ -│ │ Main Worker │ │ Temporal Client │ │ Task Queues │ │ -│ │ │◄──►│ │◄──►│ │ │ -│ │ - Metrics Server│ │ - Namespace Mgmt │ │ - predictions_batch-queue│ │ -│ │ - Notifications │ │ - Runtime Config │ │ - minimal_retrain-queue │ │ -│ │ - Lifecycle │ │ - Connection │ │ - Auto-scaling │ │ -│ │ - Health Checks │ │ - Security │ │ - Load Balancing │ │ -│ └─────────────────┘ └──────────────────┘ └─────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────┐ - │ Workflow Layer │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ PredictionsBatch│ │ Sub-Workflows │ │ - │ │ │ │ │ │ - │ │ - Data Loading │ │ - PredictionProcess │ │ - │ │ - Configuration │ │ - FormatAndExportPrediction │ │ - │ │ - Delegation │ │ - Error Handling │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ MinimalRetrain │ │ Model Management │ │ - │ │ │ │ │ │ - │ │ - Retraining │ │ - Version Control │ │ - │ │ - Validation │ │ - Production Updates │ │ - │ │ - Deployment │ │ - Quality Assurance │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - └─────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────┐ - │ Activity Layer │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ Data Quality │ │ MLFlow Operations │ │ - │ │ │ │ │ │ - │ │ - Input Gates │ │ - Model Transform │ │ - │ │ - Validation │ │ - Model Prediction │ │ - │ │ - Filtering │ │ - Response Validation │ │ - │ │ - Policy Mgmt │ │ - Error Handling │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ Storage Ops │ │ OPC Operations │ │ - │ │ │ │ │ │ - │ │ - PostgreSQL │ │ - Server Connections │ │ - │ │ - Data Export │ │ - Tag Writing │ │ - │ │ - Metrics │ │ - Real-time Export │ │ - │ │ - Cleanup │ │ - Error Recovery │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - └─────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────┐ - │ Data Services │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ PostgreSQL │ │ MongoDB │ │ - │ │ │ │ │ │ - │ │ - Predictions │ │ - Notifications │ │ - │ │ - Metadata │ │ - Audit Logs │ │ - │ │ - Metrics │ │ - Configuration │ │ - │ │ - Cleanup │ │ - User Management │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ MLFlow API │ │ OPC Servers │ │ - │ │ │ │ │ │ - │ │ - Model Serving │ │ - Real-time Data │ │ - │ │ - Transform │ │ - Industrial Integration │ │ - │ │ - Prediction │ │ - Security & Auth │ │ - │ │ - Versioning │ │ - Load Balancing │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - └─────────────────────────────────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────────────┐ - │ External Systems │ - │ ┌─────────────────┐ ┌─────────────────────────────┐ │ - │ │ Prometheus │ │ Kubernetes │ │ - │ │ │ │ │ │ - │ │ - Metrics │ │ - Orchestration │ │ - │ │ - Alerting │ │ - Scaling │ │ - │ │ - Dashboards │ │ - Health Checks │ │ - │ │ - Monitoring │ │ - Resource Management │ │ - │ └─────────────────┘ └─────────────────────────────┘ │ - └─────────────────────────────────────────────────────────┘ -``` ### Architecture Principles @@ -284,6 +195,14 @@ The **PredictionsBatch** workflow is the main entry point for batch prediction p } ``` +#### Architecture Diagram +```mermaid +flowchart LR + A[1. load_custom_query] --> B[2. prediction_process 🔃] + + A -.-> Database[(Database)] +``` + ### 2. Prediction Process Workflow (`prediction_process.py`) The **PredictionProcess** workflow implements the core prediction pipeline for ML model inference. It handles data quality validation, MLFlow model interactions, and prediction processing. @@ -345,6 +264,17 @@ The **PredictionProcess** workflow implements the core prediction pipeline for M } ``` +#### Architecture Diagram +```mermaid +flowchart LR + A[1. get_last_timestamp] --> B[2. input_gate] --> C[3. request_transform] --> D[4. mlflow_response_gate] --> E[5. mlflow_content_gate] --> F[6. request_predict] --> G[7. mlflow_response_gate] --> H[8. format_and_export_prediction🔃] + + A -.-> Redis[(Redis)] + C -.-> MLFlow[MLFlow] + F -.-> MLFlow[MLFlow] + G -.-> Filters[MLFlow Filters] +``` + ### 3. Format and Export Prediction Workflow (`format_and_export_prediction.py`) The **FormatAndExportPrediction** workflow handles prediction data formatting and export operations to multiple destinations. @@ -368,6 +298,17 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an - **Performance Monitoring**: Comprehensive metrics for export operations - **Error Handling**: Robust error handling with notification integration +#### Architecture Diagram +```mermaid +flowchart LR + A[1. format_prediction/format_default_prediction] --> B[2. write_opc_data] --> C[3. export_data_to_postgres] --> D[4. write_metrics] + + A -.-> Format[Data Formatting] + B -.-> OPC[OPC Servers] + C -.-> PostgreSQL[(PostgreSQL)] + D -.-> Prometheus[Prometheus] +``` + ### 4. Minimal Retrain Workflow (`minimal_retrain.py`) The **MinimalRetrain** workflow handles automated model retraining and production model updates. @@ -385,6 +326,17 @@ The **MinimalRetrain** workflow handles automated model retraining and productio 4. **Production Update**: Updates production model if quality criteria met 5. **Data Export**: Exports training data for analysis +#### Architecture Diagram +```mermaid +flowchart LR + A[1. load_custom_query] --> B[2. retrain_model] --> C[3. update_production_model] --> D[4. export_data_to_postgres] + + A -.-> Database[(Database)] + B -.-> MLFlow[MLFlow] + C -.-> MLFlow[MLFlow] + D -.-> PostgreSQL[(PostgreSQL)] +``` + ## 📋 Prerequisites - Python 3.11+