SIENTIAPDE-1773
Enhance environment configuration and update dependencies - Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`. - Updated `git-requirements-mapping.txt` to reflect changes in repository names. - Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1. - Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration. - Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
This commit is contained in:
27
README.md
27
README.md
@@ -163,7 +163,7 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
|
||||
#### **Data Services (`laborious/utils/`)**
|
||||
- `connectors_config.py`: Env-driven configuration builders
|
||||
- `models/minio_dataframe_payload.py`: MinIO-offloaded DataFrame payload model
|
||||
- `repository/model_repository.py`: MLFlow operations and retraining
|
||||
- ML models are loaded via `SientiaMLflowRepository` (wrapper-based, `@production` alias) constructed in `Activities` from `build_mlflow_config()`.
|
||||
- `repository/opc_repository.py`: OPC communication and writes
|
||||
- `repository/minio_manager.py`: MinIO object storage operations
|
||||
- `filters/conditional_filters.py` and `filters/mlflow_filters.py`
|
||||
@@ -730,7 +730,6 @@ tests/
|
||||
│ │ ├── test_conditional_filters.py
|
||||
│ │ └── test_mlflow_filters.py
|
||||
│ └── repository/
|
||||
│ ├── test_model_repository.py
|
||||
│ └── test_opc_repository.py
|
||||
```
|
||||
|
||||
@@ -803,7 +802,20 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
|
||||
| `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
|
||||
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
|
||||
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
|
||||
| `RUNTIME` | Plugin store runtime name installed at worker boot (required for model-loading workers) | `single` | Yes* |
|
||||
| `STORE_BASE_URL` | Plugin store Git server base URL | `http://localhost:3000` | Yes* |
|
||||
| `STORE_OWNER` | Plugin store repository owner | `sientia` | Yes* |
|
||||
| `STORE_REPO` | Plugin store repository name | `model-library-store` | Yes* |
|
||||
| `STORE_BRANCH` | Optional branch for the store repository | `main` | No |
|
||||
| `STORE_USERNAME` | HTTP username for the Git store | `None` | No |
|
||||
| `STORE_PASSWORD` | HTTP password/token for the Git store | `None` | No |
|
||||
| `STORE_CACHE_TTL_SECONDS` | Optional cache TTL for store metadata | `None` | No |
|
||||
| `PYPI_SERVER` | Private PyPI index URL for runtime wheels | `http://localhost:5000` | Yes* |
|
||||
| `PYPI_USERNAME` | Optional PyPI basic-auth username | `None` | No |
|
||||
| `PYPI_PASSWORD` | Optional PyPI basic-auth password | `None` | No |
|
||||
| `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
|
||||
|
||||
\* `RUNTIME`, PluginStore (`STORE_*`), and `PYPI_SERVER` are required for workers that install a runtime and load `SientiaModel` wrappers (`PredictionsBatch`, `MinimalRetrain`). Workers that only run drift/simple-metrics style jobs may omit them when those workflows are deployed separately.
|
||||
| `OPC_ID` | OPC server identifier | `1` | No |
|
||||
| `OPC_URL` | OPC server URL | `opc.tcp://localhost:4840` | No |
|
||||
| `OPC_SERVER_URI` | OPC server URI | `opc.tcp://localhost:4840` | No |
|
||||
@@ -1038,11 +1050,8 @@ This is the configuration created by the Orchestrator in Temporal.
|
||||
"EMPTY_DATA":{"config":{},"policy":"STOP"}
|
||||
},
|
||||
"model_config":{
|
||||
"is_compressed":true,
|
||||
"predict_flavor":"pyfunc",
|
||||
"retention_minutes":60,
|
||||
"retention_target":"artifact",
|
||||
"transform_function_keyword":"transform"
|
||||
"target":"sensor_or_label_column",
|
||||
"retention_minutes":60
|
||||
},
|
||||
"model_id":"352",
|
||||
"model_name":"courier",
|
||||
@@ -1081,8 +1090,7 @@ laborious/
|
||||
│ ├── prediction_process.py # Core prediction workflow
|
||||
│ └── format_and_export_prediction.py # Export workflow
|
||||
├── worker/ # Worker implementation
|
||||
│ ├── worker.py # Main worker orchestrator
|
||||
│ └── prepare_worker.py # Worker factory with autoscaling config
|
||||
│ └── worker.py # Entrypoint; workers built via `sientia_do.temporal.worker.prepare_worker`
|
||||
├── utils/ # Utility functions
|
||||
│ ├── connectors_config.py # Environment-driven config builders
|
||||
│ ├── models/ # Data models
|
||||
@@ -1091,7 +1099,6 @@ laborious/
|
||||
│ │ ├── conditional_filters.py # Conditional data filters
|
||||
│ │ └── mlflow_filters.py # MLFlow response filters
|
||||
│ └── repository/ # Data access layer
|
||||
│ ├── model_repository.py # MLFlow model operations
|
||||
│ ├── opc_repository.py # OPC server operations
|
||||
│ └── minio_manager.py # MinIO object storage operations
|
||||
├── metrics.py # Prometheus metrics definitions
|
||||
|
||||
Reference in New Issue
Block a user