SIENTIAPDE-1773

Enhance environment configuration and update dependencies

- Added new environment variables for PluginStore and MLflow configuration in `.env.example`, including `RUNTIME`, `STORE_BASE_URL`, `STORE_OWNER`, `STORE_REPO`, `STORE_BRANCH`, `STORE_USERNAME`, `STORE_PASSWORD`, `STORE_CACHE_TTL_SECONDS`, `PYPI_SERVER`, `PYPI_USERNAME`, and `PYPI_PASSWORD`.
- Updated `git-requirements-mapping.txt` to reflect changes in repository names.
- Modified `requirements-light.txt` and `requirements.txt` to upgrade `sientia-dataops-library` to version 1.12.0 and `sientia-mlops-library` to version 0.8.1.
- Updated `values.yaml` to include new environment variables for worker runtime and PluginStore configuration.
- Refactored E2E tests to utilize new MLflow repository stubs and PluginStore mocks for improved testing accuracy.
This commit is contained in:
vitor-aignosi
2026-05-05 16:52:51 -03:00
parent 473bd0b03f
commit 1ce8b9d3a7
23 changed files with 1280 additions and 3970 deletions

View File

@@ -163,7 +163,7 @@ Laborious uses a Temporal-based architecture with strong separation of concerns
#### **Data Services (`laborious/utils/`)**
- `connectors_config.py`: Env-driven configuration builders
- `models/minio_dataframe_payload.py`: MinIO-offloaded DataFrame payload model
- `repository/model_repository.py`: MLFlow operations and retraining
- ML models are loaded via `SientiaMLflowRepository` (wrapper-based, `@production` alias) constructed in `Activities` from `build_mlflow_config()`.
- `repository/opc_repository.py`: OPC communication and writes
- `repository/minio_manager.py`: MinIO object storage operations
- `filters/conditional_filters.py` and `filters/mlflow_filters.py`
@@ -730,7 +730,6 @@ tests/
│ │ ├── test_conditional_filters.py
│ │ └── test_mlflow_filters.py
│ └── repository/
│ ├── test_model_repository.py
│ └── test_opc_repository.py
```
@@ -803,7 +802,20 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
| `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
| `RUNTIME` | Plugin store runtime name installed at worker boot (required for model-loading workers) | `single` | Yes* |
| `STORE_BASE_URL` | Plugin store Git server base URL | `http://localhost:3000` | Yes* |
| `STORE_OWNER` | Plugin store repository owner | `sientia` | Yes* |
| `STORE_REPO` | Plugin store repository name | `model-library-store` | Yes* |
| `STORE_BRANCH` | Optional branch for the store repository | `main` | No |
| `STORE_USERNAME` | HTTP username for the Git store | `None` | No |
| `STORE_PASSWORD` | HTTP password/token for the Git store | `None` | No |
| `STORE_CACHE_TTL_SECONDS` | Optional cache TTL for store metadata | `None` | No |
| `PYPI_SERVER` | Private PyPI index URL for runtime wheels | `http://localhost:5000` | Yes* |
| `PYPI_USERNAME` | Optional PyPI basic-auth username | `None` | No |
| `PYPI_PASSWORD` | Optional PyPI basic-auth password | `None` | No |
| `OPC_CONFIG` | OPC server configuration (JSON) | `{}` | No |
\* `RUNTIME`, PluginStore (`STORE_*`), and `PYPI_SERVER` are required for workers that install a runtime and load `SientiaModel` wrappers (`PredictionsBatch`, `MinimalRetrain`). Workers that only run drift/simple-metrics style jobs may omit them when those workflows are deployed separately.
| `OPC_ID` | OPC server identifier | `1` | No |
| `OPC_URL` | OPC server URL | `opc.tcp://localhost:4840` | No |
| `OPC_SERVER_URI` | OPC server URI | `opc.tcp://localhost:4840` | No |
@@ -1038,11 +1050,8 @@ This is the configuration created by the Orchestrator in Temporal.
"EMPTY_DATA":{"config":{},"policy":"STOP"}
},
"model_config":{
"is_compressed":true,
"predict_flavor":"pyfunc",
"retention_minutes":60,
"retention_target":"artifact",
"transform_function_keyword":"transform"
"target":"sensor_or_label_column",
"retention_minutes":60
},
"model_id":"352",
"model_name":"courier",
@@ -1081,8 +1090,7 @@ laborious/
│ ├── prediction_process.py # Core prediction workflow
│ └── format_and_export_prediction.py # Export workflow
├── worker/ # Worker implementation
── worker.py # Main worker orchestrator
│ └── prepare_worker.py # Worker factory with autoscaling config
── worker.py # Entrypoint; workers built via `sientia_do.temporal.worker.prepare_worker`
├── utils/ # Utility functions
│ ├── connectors_config.py # Environment-driven config builders
│ ├── models/ # Data models
@@ -1091,7 +1099,6 @@ laborious/
│ │ ├── conditional_filters.py # Conditional data filters
│ │ └── mlflow_filters.py # MLFlow response filters
│ └── repository/ # Data access layer
│ ├── model_repository.py # MLFlow model operations
│ ├── opc_repository.py # OPC server operations
│ └── minio_manager.py # MinIO object storage operations
├── metrics.py # Prometheus metrics definitions