feat: update environment configuration and remove deprecated model serving

- Modified `.env.example` to set local defaults for PostgreSQL, MLflow, and MinIO configurations. - Added MongoDB configuration parameters to the environment setup. - Updated `README.md` to reflect changes in workflow input parameters and task queue naming conventions. - Removed the `ModelServing` class to streamline the codebase, as it was deemed unnecessary. - Adjusted `connectors_config.py` to align with new environment variable names and improve clarity. - Updated tests to reflect changes in configuration handling and removed tests related to the deleted `ModelServing` class.
2026-04-07 16:58:50 -03:00
parent c5cd382350
commit 0ae03b246f
11 changed files with 301 additions and 1303 deletions
--- a/README.md
+++ b/README.md
@@ -15,6 +15,7 @@ An enterprise-grade ML model training orchestration platform built on Temporal.
  - [Security Architecture](#security-architecture)
 - [Workflows](#workflows)
  - [Train Model Workflow](#train-model-workflow-train_modelpy)
+  - [Train model workflow input (sample)](#train-model-workflow-input-sample)
  - [Cleanup Files Workflow](#cleanup-files-workflow-cleanup_filespy)
 - [Installation & Setup](#installation--setup)
  - [Prerequisites](#prerequisites)
@@ -154,9 +155,7 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
  - Health check endpoints for Kubernetes liveness/readiness probes
  - Graceful shutdown with cleanup procedures
  - Multi-instance deployment support
-  - Two dedicated task queues:
-    - `train_model-queue`: ML model training workflows
-    - `cleanup-queue`: File cleanup workflows
+  - Task queues are derived from `RUNTIME` (default `single`): `train_model-<runtime>-queue` and `cleanup_files-<runtime>-queue` (see `prepare_worker.build_queue_name`)
  - Automated cleanup schedule management

 #### **Workflows (`model_manager/workflows/`)**
@@ -276,39 +275,13 @@ The **TrainModel** workflow orchestrates the complete ML model training pipeline
 - **Structured Logging**: Rich context in error messages for debugging
 - **Business Validation**: 10 business rules including range checks, consistency validation, and data integrity

-#### Input Parameters
-```json
-{
-  "experiment_run_id": 123,
-  "target_variable": "price",
-  "variable_columns": ["feature1", "feature2"],
-  "train_size": 80,
-  "shuffle": true,
-  "use_scaler": true,
-  "include_ar": false,
-  "bucket_name": "ml-data",
-  "file_name": "training_data.csv",
-  "line_separator": "\n",
-  "decimal_separator": ".",
-  "lag_train": {"feature1": 0, "feature2": 2},
-  "lag_val": {"feature1": 0, "feature2": 1},
-  "rem_static_win": false,
-  "static_threshold": null,
-  "low_lim": {"feature1": 0.0, "feature2": 0.0},
-  "upp_lim": {"feature1": 100.0, "feature2": 100.0},
-  "window": 10,
-  "experiment_name": "production_model_v1",
-  "removed_intervals": [],
-  "model_name": "Linear Regression",
-  "degree": 1,
-  "interaction_only": false,
-  "nan_treatment": "drop",
-  "start_date": null,
-  "end_date": null,
-  "scaler_name": "Standard Scaler",
-  "support_filters": {}
-}
-```
+#### Train model workflow input (sample)
+
+The workflow receives **one argument**: a JSON-serializable object whose keys match `TrainModelParams` (`model_manager/utils/models/train_model_params.py`). All fields are passed at the **top level** (not nested under `train_params`).
+
+A **minimal valid example** (only keys required by `TrainModelParams.from_dict`, plus a minimal `model_metadata` for `validate_business_rules`) is in **`input-sample.json`**. See also **`input-sample.md`** for SQL/MinIO notes. Optional inputs include `date_column`, `date_format`, `random_state` (defaults to `42`), `val_file_name`, and `model_id`.
+
+When starting the workflow from a Temporal client, use the same **task queue** as the worker: `train_model-<runtime>-queue` (for example `train_model-single-queue` when `RUNTIME=single`).

 #### Architecture Diagram
 ```mermaid
@@ -333,24 +306,17 @@ The workflow implements 5 different retry policies optimized for each operation
 | **No Retry** | - | - | - | 1 | Training/Validation (permanent data errors) |
 | **Database** | 2s | 20s | 2.0x | 5 | PostgreSQL updates (lock contention) |

-#### Business Validation Rules
+#### Business validation rules

-The workflow validates comprehensive business rules beyond type checking:
+`TrainModelParams.validate_business_rules()` runs after type coercion. Notable checks:

-1. **train_size**: Must be between 10-100%
-2. **variable_columns**: Cannot be empty
-3. **lag_train, lag_val**: Per-variable dictionaries with non-negative values
-4. **window**: Must be non-negative integer
-5. **low_lim/upp_lim**: Must have same keys and low < upp for each variable
-6. **target_variable**: Cannot be empty
-7. **bucket_name, file_name, experiment_name**: Cannot be empty or whitespace
-8. **degree**: Must be at least 1; must be >= 2 for Polynomial Regression
-9. **nan_treatment**: Must be one of 'drop', 'linear interpolation', 'fill linear'
-10. **scaler_name**: Must be 'Standard Scaler' or 'None'
-11. **model_name**: Must be 'Linear Regression' or 'Polynomial Regression'
-12. **Polynomial Regression requires Scaler**: Models with degree > 1 must have a scaler to prevent numerical overflow
-13. **Linear Regression requires degree 1**: Linear models must have degree = 1
-14. **static_threshold**: When `rem_static_win` is true and `static_threshold` has a value, it must be between 1 and 1000 (inclusive). If null, defaults to 1.
+1. **train_size**: Between 10 and 100 (percent).
+2. **variable_columns**: Non-empty list.
+3. **model_metadata**: Required (non-empty) for validation to succeed; may include JSON Schema definitions under `model_metadata.schemas.components.schemas` for `data_model`, `model`, and `opt_params` when you want schema validation of the corresponding kwargs.
+4. **target_variable**, **bucket_name**, **file_name**, **model_name**: Non-empty strings (no whitespace-only values).
+5. **date_format**: When set, must be an allowed frontend date format (see `validate_frontend_date_format`).
+
+Model-specific rules (for example polynomial degree and scaler requirements) live in the training stack and integration scenarios; see `docs/test-scenarios/` and `scripts/run_training_test.py` for scenario-based examples.

 ### Cleanup Files Workflow (`cleanup_files.py`)

@@ -386,10 +352,10 @@ The cleanup schedule is automatically created when the worker starts:

 | Configuration | Environment Variable | Default | Description |
 |--------------|---------------------|---------|-------------|
-| **Schedule ID** | `CLEANUP_SCHEDULE_ID` | `cleanup-files-daily` | Unique identifier for the schedule |
+| **Schedule ID** | (derived) | `cleanup-files-<runtime>-daily` | Built from `RUNTIME` in `cleanup_schedule.build_cleanup_schedule_id` |
 | **Cron Expression** | `CLEANUP_CRON` | `0 0 * * *` | Daily at midnight UTC |
 | **Timezone** | `CLEANUP_TIMEZONE` | `UTC` | Timezone for cron execution |
-| **Task Queue** | `CLEANUP_TASK_QUEUE` | `cleanup-queue` | Dedicated task queue |
+| **Task Queue** | (derived) | `cleanup_files-<runtime>-queue` | Must match the cleanup worker queue (`build_queue_name('CleanupFiles', runtime)`) |
 | **Execution Timeout** | `CLEANUP_EXECUTION_TIMEOUT_HOURS` | `1` | Maximum execution time (hours) |
 | **Retention Period** | `CLEANUP_RETENTION_HOURS` | `24` | Files older than this are deleted |
 | **Dry Run** | `CLEANUP_DRY_RUN` | `false` | Test mode without actual deletion |
@@ -739,6 +705,8 @@ fi
 python -m model_manager.worker.worker
 ```

+To start a **`train_model`** run from your own Temporal client, use the payload shape in **`input-sample.json`** (task queue `train_model-<runtime>-queue`, matching `RUNTIME` on the worker). For scripted tests that use the JSON scenarios under `docs/test-scenarios/`, see **`scripts/run_training_test.py`**.
+
 ## Code Quality & Validation

 ### Overview
@@ -1003,13 +971,10 @@ Test scenarios are defined as JSON files in `docs/test-scenarios/`. Each scenari
 3. Save with a descriptive name: `XX-description.json`
 4. Run with: `python scripts/run_training_test.py --scenario XX-description`

-#### Important Validations
+#### Important validations

-The training workflow enforces several business rules:
-
- **Polynomial Regression requires Scaler**: Models with `degree > 1` must have `useScaler: true` and a valid `scalerName` to prevent numerical overflow
- **Static Window Removal requires DatetimeIndex**: Scenarios with `remStaticWin: true` require data with a timestamp column for the `TimeSeriesDiscontinuityAnalyzer`
- **Variable Limits Consistency**: `lowLim` and `uppLim` must have matching keys, and `lowLim[key] < uppLim[key]` for all variables
+- **Workflow payload** (`input-sample.json`, Temporal `execute_workflow`): snake_case fields validated by `TrainModelParams` (see [Business validation rules](#business-validation-rules) above).
+- **Integration scenarios** (`docs/test-scenarios/*.json`): camelCase UI-oriented fields consumed by `scripts/run_training_test.py`, which maps them into `TrainModelParams` before running. Additional rules apply there (for example polynomial degree and scaler requirements, static window removal, variable limits); see scenario descriptions in the table above.

 ## Monitoring and Metrics

@@ -1044,49 +1009,65 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa

 ### Environment Variables

+Values are read in `model_manager/utils/connectors_config.py` and `model_manager/worker/worker.py`. Defaults below match the code.
+
 | Variable | Description | Default | Required |
 |----------|-------------|---------|----------|
-| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
+| `TEMPORAL_HOST` | Temporal server address (`host:port`) | `localhost:7233` | Yes |
 | `TEMPORAL_NAMESPACE` | Temporal namespace | `model-manager` | No |
-| `TEMPORAL_USE_TLS` | Enable TLS for Temporal connection | `false` | No |
-| `TRAIN_TASK_QUEUE` | Task queue for training workflows | `train_model-queue` | No |
-| `CLEANUP_TASK_QUEUE` | Task queue for cleanup workflows | `cleanup-queue` | No |
+| `TEMPORAL_USE_TLS` | Use TLS for Temporal gRPC (`true`/`false`). Set `true` when the endpoint serves TLS or you get HTTP redirects (for example 308) to HTTPS | `false` | No |
+| `RUNTIME` | Suffix for worker task queues (`train_model-<runtime>-queue`, `cleanup_files-<runtime>-queue`) | `single` (via `_get_runtime`) | No |
+| `TRAIN_TASK_QUEUE` | Used by **clients** (for example `scripts/run_training_test.py`), not by the worker process | unset | No |
+| `CLEANUP_TASK_QUEUE` | Used by **clients** (for example `scripts/run_cleanup_test.py`), not by the worker | unset | No |
 | `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
 | `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
 | `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
 | `POSTGRES_PASSWORD` | PostgreSQL password | `sientia` | Yes |
 | `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
-| `POSTGRES_MIN_CONNECTIONS` | Minimum PostgreSQL connections | `5` | No |
-| `POSTGRES_MAX_CONNECTIONS` | Maximum PostgreSQL connections | `20` | No |
-| `MLFLOW_URL` | MLFlow server URL (full URL with protocol and port) | `http://localhost:5080` | Yes |
-| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
-| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
-| `MINIO_ENDPOINT_URL` | MinIO server endpoint | `http://minio.minio.svc.cluster.local:9000` | Yes |
+| `POSTGRES_MIN_CONNECTIONS` | Minimum pool size | `5` | No |
+| `POSTGRES_MAX_CONNECTIONS` | Maximum pool size | `20` | No |
+| `MLFLOW_URL` | MLflow tracking URL (scheme, host, and port) | `http://localhost:5080` | Yes |
+| `MLFLOW_USERNAME` | MLflow basic auth username | `aignosi` | Yes |
+| `MLFLOW_PASSWORD` | MLflow basic auth password | `aignosi` | Yes |
+| `MINIO_ENDPOINT_URL` | MinIO / S3 endpoint URL | `http://localhost:9000` | Yes |
 | `MINIO_ACCESS_KEY` | MinIO access key | `minioadmin` | Yes |
 | `MINIO_SECRET_KEY` | MinIO secret key | `minioadmin` | Yes |
 | `MINIO_REGION` | MinIO region | `us-east-1` | No |
-| `MINIO_USE_SSL` | Enable SSL for MinIO | `false` | No |
-| `MINIO_MAX_RETRY_ATTEMPTS` | Maximum retry attempts | `3` | No |
-| `MINIO_RETRY_MODE` | Retry mode (standard/adaptive) | `adaptive` | No |
+| `MINIO_SECURE` | Use TLS for MinIO client (`true`/`false`) | `false` | No |
+| `MINIO_DEFAULT_BUCKET` | Default bucket for `MinioRepository` | `model-training` | No |
+| `MINIO_MAX_RETRY_ATTEMPTS` | S3 retry attempts | `3` | No |
+| `MINIO_RETRY_MODE` | Retry mode | `adaptive` | No |
 | `MINIO_CONNECT_TIMEOUT` | Connection timeout (seconds) | `10` | No |
 | `MINIO_READ_TIMEOUT` | Read timeout (seconds) | `60` | No |
-| `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
+| `MONGODB_URL` | MongoDB host:port (no scheme; used inside connection string) | `localhost:27018` | Yes |
 | `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
 | `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
 | `MONGODB_DATABASE` | MongoDB database name | `sientia` | Yes |
-| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index hours | `1` | No |
-| `CLEANUP_SCHEDULE_ID` | Cleanup schedule identifier | `cleanup-files-daily` | No |
-| `CLEANUP_CRON` | Cleanup cron expression | `0 0 * * *` | No |
+| `MONGODB_TTL_INDEX_HOURS` | TTL index duration (hours) | `1` | No |
+| `STORE_BASE_URL` | Plugin store Git server base URL | `http://localhost:3000` | No |
+| `STORE_OWNER` | Git owner/org | `sientia` | No |
+| `STORE_REPO` | Git repository | `model-library-store` | No |
+| `STORE_BRANCH` | Optional branch | unset | No |
+| `STORE_USERNAME` / `STORE_PASSWORD` | Git HTTP credentials | unset | No |
+| `STORE_CACHE_TTL_SECONDS` | Plugin index cache TTL | unset | No |
+| `PYPI_SERVER` | Custom PyPI index URL | `http://localhost:5000` | No |
+| `PYPI_USERNAME` / `PYPI_PASSWORD` | PyPI credentials | unset | No |
+| `CLEANUP_CRON` | Cleanup schedule cron | `0 0 * * *` | No |
 | `CLEANUP_TIMEZONE` | Cleanup schedule timezone | `UTC` | No |
-| `CLEANUP_EXECUTION_TIMEOUT_HOURS` | Cleanup execution timeout | `1` | No |
-| `CLEANUP_RETENTION_HOURS` | File retention period (hours) | `24` | No |
-| `CLEANUP_DRY_RUN` | Dry-run mode (no actual deletion) | `false` | No |
-| `LOG_LEVEL` | Application log level | `INFO` | No |
-| `PROJECT_NAME` | Project name for metrics | `model-manager` | No |
+| `CLEANUP_EXECUTION_TIMEOUT_HOURS` | Cleanup workflow timeout (hours) | `1` | No |
+| `CLEANUP_RETENTION_HOURS` | Local temp retention (hours) | `24` | No |
+| `CLEANUP_DRY_RUN` | Cleanup dry-run | `false` | No |
+| `LOG_LEVEL` | Log level | `INFO` | No |
+| `PROJECT_NAME` | Project name for notifications/metrics | `model-manager` | No |
 | `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
 | `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
-| `POD_ID` | Kubernetes pod identifier | `None` | No |
-| `EXTRA_PIP_REQUIREMENTS` | Extra pip requirements for MLFlow model serving | `None` | No |
+| `POD_ID` | Pod label for metrics | unset | No |
+| `EXTRA_PIP_REQUIREMENTS` | Extra `pip` packages for runtime installs | unset | No |
+| `TIMEOUT_VALIDATE_PARAMS` | Activity timeout (seconds) | `30` | No |
+| `TIMEOUT_TRAIN_MODEL` | Training activity timeout (seconds) | `2700` | No |
+| `TIMEOUT_DELETE_FILE` | Delete/cleanup activity timeout (seconds) | `120` | No |
+| `TIMEOUT_UPDATE_DATABASE` | DB update activity timeout (seconds) | `30` | No |
+| `TIMEOUT_CLEANUP_LOCAL` | Cleanup workflow activity timeout (seconds) | `120` | No |

 #### Workflow Activity Timeouts

@@ -1098,6 +1079,7 @@ These timeouts control how long each activity in workflows can run before timing
 |----------|-------------|---------|-------------------|
 | `TIMEOUT_VALIDATE_PARAMS` | Parameter validation timeout | `30` | Fast operation, no I/O |
 | `TIMEOUT_TRAIN_MODEL` | Model training timeout | `2700` | Large dataset processing (45 min) |
+| `TIMEOUT_DELETE_FILE` | File delete / related I/O timeout | `120` | Network storage |
 | `TIMEOUT_UPDATE_DATABASE` | Database update timeout | `30` | PostgreSQL update query (30 sec) |

 **Cleanup Workflow Timeouts:**
@@ -1246,6 +1228,7 @@ sientia-dataops-model-manager/
 ├── requirements-dev.txt     # Development dependencies
 ├── validate.sh              # Code quality validation script
 ├── run_local.sh             # Local execution script
+├── input-sample.json        # Example payload for the train_model workflow
 └── README.md                # This file
 ```

@@ -1280,22 +1263,23 @@ sientia-dataops-model-manager/

 ### Common Issues

-1. **Temporal Connection Failures**
-   - Verify Temporal server is running and accessible
+1. **Temporal connection failures**
+   - Verify Temporal server is running and reachable at `TEMPORAL_HOST`
   - Check namespace configuration and permissions
   - Review server logs for connection issues
+   - If you see **308 Permanent Redirect** or **invalid compression flag** on connect, the endpoint likely expects **TLS** while `TEMPORAL_USE_TLS` is `false`. Set `TEMPORAL_USE_TLS=true` and point `TEMPORAL_HOST` at the correct TLS gRPC address (host and port depend on your ingress or load balancer)

-2. **MLFlow Connection Issues**
+2. **MLFlow connection issues**
   - Verify MLFlow server is running and accessible
   - Check authentication credentials and permissions
   - Ensure model names and versions exist

-3. **Database Connection Issues**
+3. **Database connection issues**
   - Verify PostgreSQL service is running
   - Check connection credentials and network access
   - Ensure proper connection pool configuration

-4. **Workflow Execution Failures**
+4. **Workflow execution failures**
   - Review activity error logs and notifications
   - Check training parameter validation errors
   - Verify input data format and required fields
@@ -1321,9 +1305,7 @@ export LOG_LEVEL=DEBUG
 ### Scaling Considerations

 - **Horizontal Scaling**: Deploy multiple worker instances
- **Task Queue Distribution**: Two dedicated task queues for workflow isolation
-  - `train_model-queue`: Training workflows
-  - `cleanup-queue`: Cleanup workflows
+- **Task queue distribution**: Workers register `train_model-<runtime>-queue` and `cleanup_files-<runtime>-queue` (see `RUNTIME`)
 - **Database Performance**: Optimize indexes and connection pooling
 - **MLFlow Performance**: Configure appropriate model serving resources
 - **Storage Management**: Adjust cleanup retention period based on storage capacity and costs