SIENTIAPDE-1273

Enhance data handling and export processes in Laborious workflows

- Updated `gates.py` to improve data quality validation, filtering, and formatting operations, including enhanced metrics recording.
- Refined `mlflow.py` to better manage model transformations and reference data retrieval from MLflow Model Registry.
- Enhanced `format_and_export_prediction.py` to support separate export of transformed data, improving flexibility in data handling.
- Added comprehensive test coverage for new functionalities, including transformed data formatting and retrain report generation.
- Improved documentation in `README.md` to reflect changes in activities and workflows, ensuring clarity on data processing and export paths.

README.md

@@ -130,8 +130,14 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
 - `minimal_retrain.py`: Automated model retraining and production update
 
 #### **Activities (`laborious/activities/`)**
-- `gates.py`: Data quality validation and filtering
-- `mlflow.py`: Transform and predict operations
+- `gates.py`: Data quality validation, filtering, and data formatting operations
+  - Input/response/content gates for quality validation
+  - Prediction and transformed data formatting
+  - Retrain report formatting and metrics recording
+- `mlflow.py`: Transform, predict, and model management operations
+  - MLFlow model transformation and prediction
+  - Model retraining and production updates
+  - Reference data retrieval from MLflow Model Registry
 - `opc.py`: OPC UA export to industrial systems (optional)
 - `activities.py`: Aggregates activity interfaces
 
@@ -146,7 +152,9 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
 #### **1. Batch Prediction Pipeline**
 ```
 Input Data (PostgreSQL) → Data Quality Gates → MLFlow Transform →
-MLFlow Prediction → Response Validation → Export (PostgreSQL [+ OPC])
+MLFlow Prediction → Response Validation → Format & Export
+  ├─→ Predictions → PostgreSQL [+ OPC]
+  └─→ Transformed Data → PostgreSQL (optional)
 ```
 
 #### **2. Model Retraining Pipeline**
@@ -318,27 +326,37 @@ The **FormatAndExportPrediction** workflow handles prediction data formatting an
 #### Execution Flow
 1. **Path Decision**: Determines formatting path based on configuration
 2. **Data Formatting**: Formats prediction data for specific output requirements
-3. **PostgreSQL Export**: Writes formatted predictions to database
-4. **OPC Export**: Writes predictions to OPC servers
-5. **Metrics Recording**: Records export performance and success metrics
+3. **Transformed Data Processing**: Optionally formats and exports transformed data separately
+4. **PostgreSQL Export**: Writes formatted predictions to database
+5. **OPC Export**: Writes predictions to OPC servers
+6. **Metrics Recording**: Records export performance and success metrics
 
 #### Key Features
 - **Flexible Formatting**: Configurable output formats for different destinations
 - **Multi-Destination Export**: PostgreSQL and OPC server integration
+- **Transformed Data Export**: Optional separate export of MLFlow transformed data
 - **Performance Monitoring**: Comprehensive metrics for export operations
 - **Error Handling**: Robust error handling with notification integration
 
 #### Architecture Diagram
 ```mermaid
 flowchart LR
-    A[1. format_prediction/format_default_prediction] --> B[2. write_opc_data] --> C[3. export_data_to_postgres] --> D[4. write_metrics]
+    A[1. format_prediction/format_default_prediction] --> B[2. format_transformed_data] --> C[3. write_opc_data] --> D[4. export_data_to_postgres] --> E[5. write_metrics]
     
     A -.-> Format[Data Formatting]
-    B -.-> OPC[OPC Servers]
-    C -.-> PostgreSQL[(PostgreSQL)]
-    D -.-> Prometheus[Prometheus]
+    B -.-> Transform[Transformed Data]
+    C -.-> OPC[OPC Servers]
+    D -.-> PostgreSQL[(PostgreSQL)]
+    E -.-> Prometheus[Prometheus]
 ```
 
+#### Transformed Data Export
+When `transformed_data` is provided in the input, the workflow will:
+- Format the transformed data using `format_transformed_data` activity
+- Export it to a separate table (`transform_table_name`) asynchronously
+- Wait for both prediction and transformed data exports to complete
+- This enables separate tracking of model transformations for analysis and debugging
+
 ### 4. Minimal Retrain Workflow (`minimal_retrain.py`)
 
 The **MinimalRetrain** workflow handles automated model retraining and production model updates.
@@ -579,11 +597,24 @@ The workflow at `.github/workflows/quality-gate.yml` executes validations on eac
 ```
 tests/
 ├── activities/           # Activity implementation tests
-├── workflow/            # Workflow orchestration tests
+│   ├── test_gates.py    # Data quality gates and formatting tests
+│   ├── test_mlflow.py   # MLFlow operations and reference data tests
+│   └── ...              # Other activity tests
+├── workflows/           # Workflow orchestration tests
+│   └── subworkflows/    # Sub-workflow tests
+│       └── test_format_and_export_prediction.py  # Export workflow tests
 ├── utils/               # Utility function tests
 └── integration/         # End-to-end workflow tests
 ```
 
+### Test Coverage
+The test suite provides comprehensive coverage for:
+- **Data Quality Gates**: Input, response, and content validation filters
+- **Data Formatting**: Prediction, transformed data, and retrain report formatting
+- **MLFlow Operations**: Transform, predict, retrain, and reference data retrieval
+- **Workflow Orchestration**: Complete workflow execution paths and error handling
+- **Metrics Recording**: Performance monitoring and OPC export metrics
+
 ### Test Execution
 ```bash
 # Install test dependencies