SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations.

README.md

@@ -39,6 +39,12 @@ An enterprise-grade ML model training orchestration platform built on Temporal.
 - [Testing](#testing)
   - [Test Structure](#test-structure)
   - [Test Execution](#test-execution)
+  - [Integration Tests](#integration-tests)
+    - [Running Integration Tests](#running-integration-tests)
+    - [Test Scenarios](#test-scenarios)
+    - [Scenario File Structure](#scenario-file-structure)
+    - [Creating New Scenarios](#creating-new-scenarios)
+    - [Important Validations](#important-validations)
 - [Monitoring and Metrics](#monitoring-and-metrics)
   - [Application Health Metrics](#application-health-metrics)
   - [Training Metrics](#training-metrics)
@@ -871,6 +877,88 @@ pytest tests/activities/test_training.py
 pytest tests/workflows/test_train_model.py
 ```
 
+### Integration Tests
+
+The project includes integration tests that validate the complete training workflow against a running Temporal cluster. These tests use JSON-based scenario files for easy configuration and maintenance.
+
+#### Running Integration Tests
+
+```bash
+# Run a specific test scenario
+python scripts/run_training_test.py --scenario 01-linear-regression-basic
+
+# Run with custom data file
+python scripts/run_training_test.py --scenario 03-polynomial-regression-degree2 --data-file /path/to/data.csv
+
+# List available scenarios
+ls docs/test-scenarios/
+```
+
+#### Test Scenarios
+
+Test scenarios are defined as JSON files in `docs/test-scenarios/`. Each scenario configures a complete training workflow with specific parameters:
+
+| Scenario | Description | Key Features |
+|----------|-------------|--------------|
+| `01-linear-regression-basic` | Basic linear regression | No scaler, no lags |
+| `02-linear-regression-with-scaler` | Linear regression with normalization | Standard Scaler enabled |
+| `03-polynomial-regression-degree2` | Polynomial regression (degree 2) | Requires scaler (mandatory) |
+| `04-polynomial-regression-degree3` | Polynomial regression (degree 3) | Requires scaler (mandatory) |
+| `05-linear-regression-with-lags` | Linear regression with lag features | Lag train/val configuration |
+| `06-linear-regression-nan-interpolation` | Linear regression with NaN handling | `nanTreatment: "interpolate"` |
+| `07-linear-regression-static-window-removal` | Linear regression with static window removal | `remStaticWin: true` |
+| `08-linear-regression-with-limits` | Linear regression with variable limits | `lowLim`/`uppLim` configuration |
+| `09-polynomial-degree2-with-scaler-and-lags` | Complete polynomial scenario | Scaler + lags + degree 2 |
+| `10-linear-regression-with-ar` | Linear regression with autoregressive variable | `includeAr: true` |
+
+#### Scenario File Structure
+
+```json
+{
+  "_description": "Human-readable description of the scenario",
+  "experimentName": "test-experiment-name",
+  "username": "user@example.com",
+  "modelName": "Linear Regression",
+  "targetVariable": "target_column_name",
+  "variableColumns": ["feature1", "feature2"],
+  "lagTrain": {"feature1": 0, "feature2": 0},
+  "lagVal": {"feature1": 0, "feature2": 0},
+  "remStaticWin": false,
+  "lowLim": {},
+  "uppLim": {},
+  "window": 0,
+  "useScaler": false,
+  "includeAr": false,
+  "trainSize": 80,
+  "shuffle": true,
+  "lineSeparator": ",",
+  "decimalSeparator": ".",
+  "removedIntervals": [],
+  "degree": 1,
+  "interactionOnly": false,
+  "nanTreatment": "drop",
+  "startDate": null,
+  "endDate": null,
+  "scalerName": "None",
+  "supportFilters": {}
+}
+```
+
+#### Creating New Scenarios
+
+1. Copy an existing scenario file as a template
+2. Modify parameters according to your test case
+3. Save with a descriptive name: `XX-description.json`
+4. Run with: `python scripts/run_training_test.py --scenario XX-description`
+
+#### Important Validations
+
+The training workflow enforces several business rules:
+
+- **Polynomial Regression requires Scaler**: Models with `degree > 1` must have `useScaler: true` and a valid `scalerName` to prevent numerical overflow
+- **Static Window Removal requires DatetimeIndex**: Scenarios with `remStaticWin: true` require data with a timestamp column for the `TimeSeriesDiscontinuityAnalyzer`
+- **Variable Limits Consistency**: `lowLim` and `uppLim` must have matching keys, and `lowLim[key] < uppLim[key]` for all variables
+
 ## Monitoring and Metrics
 
 The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring: