SIENTIAPDE-1645: Overhaul deployment, add Claude AI guidance, and enhance CI/CD.
Migrated to a shared sientia-module image and umbrella Helm chart, removing the standalone Dockerfile, .dockerignore, and values.yaml. Introduced CLAUDE.md for AI-assisted development. Updated quality-gate.yml with push triggers, concurrency, and refined permissions. Bumped project version to 2.0.0 to reflect these architectural changes.
This commit is contained in:
@@ -1,93 +0,0 @@
|
|||||||
# ============================================================================
|
|
||||||
# WHITELIST APPROACH: Block everything by default, then allow only what's needed
|
|
||||||
# ============================================================================
|
|
||||||
|
|
||||||
# Block everything first
|
|
||||||
*
|
|
||||||
|
|
||||||
# ============================================================================
|
|
||||||
# ALLOW: Application source code (model_manager package)
|
|
||||||
# ============================================================================
|
|
||||||
|
|
||||||
# Allow the main package directory and all Python files
|
|
||||||
!model_manager/
|
|
||||||
!model_manager/**/*.py
|
|
||||||
!model_manager/**/__init__.py
|
|
||||||
|
|
||||||
# Allow subdirectories structure
|
|
||||||
!model_manager/activities/
|
|
||||||
!model_manager/activities/**
|
|
||||||
!model_manager/schedules/
|
|
||||||
!model_manager/schedules/**
|
|
||||||
!model_manager/sientia/
|
|
||||||
!model_manager/sientia/**
|
|
||||||
!model_manager/utils/
|
|
||||||
!model_manager/utils/**
|
|
||||||
!model_manager/utils/models/
|
|
||||||
!model_manager/utils/models/**
|
|
||||||
!model_manager/utils/repository/
|
|
||||||
!model_manager/utils/repository/**
|
|
||||||
!model_manager/worker/
|
|
||||||
!model_manager/worker/**
|
|
||||||
!model_manager/workflows/
|
|
||||||
!model_manager/workflows/**
|
|
||||||
|
|
||||||
# Allow reports directory with header.html
|
|
||||||
!model_manager/reports/
|
|
||||||
!model_manager/reports/header.html
|
|
||||||
|
|
||||||
# Allow temp directory structure (but not its contents)
|
|
||||||
!model_manager/reports/temp/
|
|
||||||
|
|
||||||
# ============================================================================
|
|
||||||
# ALLOW: Dependencies file (needed for pip install in Dockerfile)
|
|
||||||
# ============================================================================
|
|
||||||
!requirements.txt
|
|
||||||
|
|
||||||
# ============================================================================
|
|
||||||
# BLOCK: Explicitly block unwanted files even if they match above patterns
|
|
||||||
# ============================================================================
|
|
||||||
|
|
||||||
# Python cache and compiled files
|
|
||||||
**/__pycache__/
|
|
||||||
**/*.pyc
|
|
||||||
**/*.pyo
|
|
||||||
**/*.pyd
|
|
||||||
**/.Python
|
|
||||||
**/*.so
|
|
||||||
**/*.egg
|
|
||||||
**/*.egg-info/
|
|
||||||
|
|
||||||
# Tests (not needed in production)
|
|
||||||
model_manager/**/test_*.py
|
|
||||||
model_manager/**/*_test.py
|
|
||||||
|
|
||||||
# IDE and editor files
|
|
||||||
**/.vscode/
|
|
||||||
**/.idea/
|
|
||||||
**/*.swp
|
|
||||||
**/*.swo
|
|
||||||
**/*~
|
|
||||||
|
|
||||||
# OS files
|
|
||||||
**/.DS_Store
|
|
||||||
**/Thumbs.db
|
|
||||||
|
|
||||||
# Logs and temporary files
|
|
||||||
**/*.log
|
|
||||||
**/*.tmp
|
|
||||||
**/*.temp
|
|
||||||
|
|
||||||
# Local configuration
|
|
||||||
**/.env
|
|
||||||
**/.env.local
|
|
||||||
**/*.local
|
|
||||||
|
|
||||||
# Documentation inside code
|
|
||||||
**/*.md
|
|
||||||
**/README*
|
|
||||||
|
|
||||||
# Backup files
|
|
||||||
**/*.bak
|
|
||||||
**/*.backup
|
|
||||||
**/*.old
|
|
||||||
14
.github/workflows/quality-gate.yml
vendored
14
.github/workflows/quality-gate.yml
vendored
@@ -1,6 +1,11 @@
|
|||||||
name: Quality gate
|
name: Quality gate
|
||||||
|
|
||||||
on:
|
on:
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- main
|
||||||
|
- 'release/**'
|
||||||
|
- 'feature/**'
|
||||||
pull_request:
|
pull_request:
|
||||||
branches:
|
branches:
|
||||||
- main
|
- main
|
||||||
@@ -8,10 +13,17 @@ on:
|
|||||||
- 'feature/**'
|
- 'feature/**'
|
||||||
types: [ opened, synchronize, reopened ]
|
types: [ opened, synchronize, reopened ]
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: ${{ github.workflow }}-${{ github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
quality-gate:
|
quality-gate:
|
||||||
uses: Aignosi/github_workflow_templates/.github/workflows/python-quality-gate.yml@main
|
uses: Aignosi/github_workflow_templates/.github/workflows/python-quality-gate.yml@main
|
||||||
permissions: write-all
|
permissions:
|
||||||
|
contents: read
|
||||||
|
pull-requests: write
|
||||||
|
issues: write
|
||||||
with:
|
with:
|
||||||
project_name: 'model_manager'
|
project_name: 'model_manager'
|
||||||
repositories: 'sientia-dataops-library,sientia-model-library'
|
repositories: 'sientia-dataops-library,sientia-model-library'
|
||||||
|
|||||||
78
CLAUDE.md
Normal file
78
CLAUDE.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
Enterprise ML model training orchestration platform built on **Temporal**. Orchestrates training workflows (validation → train → save to MLFlow → cleanup), backed by PostgreSQL (experiment tracking), MinIO (data/artifacts), MongoDB (notifications), and a Git-based "Plugin Store" that supplies model runtimes/wrappers dynamically. Python 3.11+.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Local run (loads .env, starts worker)
|
||||||
|
./run_local.sh
|
||||||
|
# or manually:
|
||||||
|
python -m model_manager.worker.worker
|
||||||
|
|
||||||
|
# Full validation (format check, lint, mypy, bandit, tests w/ coverage) — run before committing
|
||||||
|
./validate.sh
|
||||||
|
./validate.sh --fix # auto-fix format/lint, no tests
|
||||||
|
./validate.sh --no-tests # skip tests
|
||||||
|
./validate.sh --only-tests # tests only
|
||||||
|
|
||||||
|
# Individual tools
|
||||||
|
ruff format --check model_manager/ tests/
|
||||||
|
ruff format model_manager/ tests/ # auto-fix
|
||||||
|
ruff check model_manager/ tests/
|
||||||
|
ruff check --fix model_manager/ tests/
|
||||||
|
mypy model_manager/
|
||||||
|
bandit -r model_manager/ -ll
|
||||||
|
|
||||||
|
# Tests
|
||||||
|
pytest # runs tests/ and e2e/ (see pyproject testpaths)
|
||||||
|
pytest tests/ # unit tests only
|
||||||
|
pytest tests/activities/test_training.py -v # single file
|
||||||
|
pytest tests/activities/test_training.py::test_name -v # single test
|
||||||
|
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=html
|
||||||
|
|
||||||
|
# Manual integration/E2E scripts (require live Temporal/Postgres/MinIO stack; run via IDE "Run Cell" or directly)
|
||||||
|
python scripts/run_training_test.py # supports --all for batch scenario execution
|
||||||
|
python scripts/run_cleanup_test.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Coverage gate in CI is 80% minimum; this repo targets 100% in practice — don't add uncovered code paths without tests.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Temporal layering
|
||||||
|
- **Worker** (`model_manager/worker/worker.py`): entrypoint. Builds config from env, connects Postgres/MLFlow/MinIO/Plugin Store, installs the `RUNTIME` model runtime via `PluginStore.install_runtime`, starts two Temporal workers (one per task queue) plus Prometheus metrics server and the cleanup schedule. All heavy imports are wrapped in `workflow.unsafe.imports_passed_through()` — required because Temporal workflow sandboxing forbids non-deterministic imports at module scope outside that block.
|
||||||
|
- **Workflows** (`model_manager/workflows/`): pure orchestration, no I/O beyond `execute_activity_method` calls.
|
||||||
|
- `train_model.py` (`TrainModel`, workflow name `train_model`): validate `experiment_run_id` → `load_model_metadata` → `validate_train_params` → update DB status `ORCHESTRATOR_WAITING_PROC` → `train_model` activity → update DB status `TRAINING_SUCCESS`/`TRAINING_ERROR` → `cleanup_resources` (always runs in `finally` if a `run_dir` exists, even on failure). Uses distinct `RetryPolicy`s per operation class: `network_retry_policy` (transient I/O), `no_retry_policy` (training/validation — data errors are permanent, and retrying would duplicate MLflow writes), `database_retry_policy` (Postgres). Timeouts are env-driven (`TIMEOUT_*`), extend `TIMEOUT_TRAIN_MODEL` instead of adding retries for slow training.
|
||||||
|
- `cleanup_files.py` (`CleanupFiles`): scheduled (`cleanup_schedule.py`, default daily at midnight UTC via `CLEANUP_CRON`) deletion of stale local temp dirs matching `name_YYYYMMDD_HHMMSS_microseconds`, older than `CLEANUP_RETENTION_HOURS`.
|
||||||
|
- Task queues are derived from `RUNTIME` env var (default `single`) via `prepare_worker.build_queue_name`: `train_model-<runtime>-queue`, `cleanup_files-<runtime>-queue`. A Temporal client starting a workflow must target the matching queue.
|
||||||
|
- **Activities** (`model_manager/activities/`): `Activities` (`activities.py`) combines `ExperimentTracking` + `Training` + `Cleanup` via multiple inheritance into one class registered with the worker — mind MRO and `__init__`/`__del__` ordering when touching this class (see the defensive `hasattr(self, 'engine')` check in `__del__`).
|
||||||
|
- `experiment_tracking.py`: single `update_experiment_run()` entrypoint handling three update kinds via `UpdateType`: `STATUS`, `STATUS_WITH_ERROR`, `MODEL_SAVED`. Error messages truncated to 1024 chars before persisting.
|
||||||
|
- `training.py`: single `train_model()` entrypoint. Receives pre-downloaded data as `BytesIO` (not re-downloaded per-activity) to avoid memory leaks; never raises on training failure — returns a result/error so the workflow can update DB status without an unhandled exception mid-activity.
|
||||||
|
- `cleanup.py`: local-filesystem-only cleanup (`cleanup_temp_directories`), no MinIO involved — MinIO artifacts are managed externally.
|
||||||
|
|
||||||
|
### Data/config layer
|
||||||
|
- `model_manager/utils/connectors_config.py`: single source of truth for env-var → config-dict builders (`build_postgres_config`, `build_mlflow_config`, `build_minio_config`, `build_mongodb_config`, `build_plugin_store_config`). Read this file (and `worker.py`) before trusting any env var default — the README table mirrors it but this file is authoritative.
|
||||||
|
- `model_manager/utils/models/train_model_params.py`: `TrainModelParams` — the workflow's single input contract. `from_dict` does type coercion; `validate_business_rules()` runs after, enforcing 7 rules (train_size range, non-empty variable_columns, required model_metadata, JSON-Schema validation of `data_model_kwargs`/`model_kwargs`/`opt_params` against schemas in `model_metadata` via `Draft202012Validator`, required non-blank strings, allowed `date_format` values, numeric `experiment_run_id`). Extending training params means updating this file's validation, not just the workflow.
|
||||||
|
- `model_manager/utils/repository/data_manager_repository.py`: data loading, feature prep, metrics, report generation — the actual ML/data logic activities delegate into.
|
||||||
|
- `model_manager/sientia/`: RCE drift metrics (`metrics.py`), Evidently-based HTML report generation (`reports.py`), custom exceptions.
|
||||||
|
- `model_manager/metrics.py`: Prometheus metric definitions (`app_up`, `workflow_execution_total`, `activity_execution_total`, training/cleanup/RCE-drift gauges). New metrics go here, not inline in activities.
|
||||||
|
|
||||||
|
### External dependencies
|
||||||
|
Two private Aignosi packages carry non-trivial logic: `sientia_do` (observability/logger, notification handler, metrics controller, Postgres/MinIO repositories) and `sientia_model` (Plugin Store, MLflow repository, model wrappers). Both are pinned in `requirements.txt`/`requirements-local.txt`; when behavior looks like it lives outside this repo, it's likely in one of these.
|
||||||
|
|
||||||
|
### Test bootstrap quirk
|
||||||
|
`tests/conftest.py` stubs `evidently` and parts of `sientia_do` at `pytest_configure` time so unit tests can import `model_manager.sientia.reports` without the full (heavy/optional) Evidently install. If you add new imports from these packages in production code paths that unit tests exercise, you likely need to extend the stubs here too.
|
||||||
|
|
||||||
|
### Test layout
|
||||||
|
- `tests/` — unit tests, mirrors `model_manager/` package structure (`tests/activities/`, `tests/schedules/`, `tests/sientia/`, etc.).
|
||||||
|
- `e2e/` — integration tests against a real Temporal/Postgres/MinIO/MongoDB stack (via `testcontainers`), driven by JSON scenario files in `docs/test-scenarios/` (see `e2e/scenarios.md` for the scenario→test mapping). Scenario payloads are snake_case, shaped like `TrainModelParams`/`input-sample.json`.
|
||||||
|
- New scenario: copy an existing `docs/test-scenarios/*.json`, add/extend a test in `e2e/test_train_model_workflow.py`, update `e2e/scenarios.md`.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
- Ruff: line-length 100, single quotes, `select = [E,W,F,I,B,C4,UP,N,YTT,S,BLE,A,C90]`, max complexity 15. `N802`/`N806` ignored (Temporal decorators use non-lowercase names). `S101`/`S105`/`S106` ignored in tests.
|
||||||
|
- mypy: `disallow_untyped_defs = false` but `check_untyped_defs = true` — existing untyped defs are tolerated, but code inside them is still checked. Several external packages (`temporalio`, `mlflow`, `sientia_model`, etc.) have `ignore_missing_imports`; `model_repository.py`/`data_manager_repository.py` have `ignore_errors = true` (legacy — don't extend that list without reason).
|
||||||
78
Dockerfile
78
Dockerfile
@@ -1,78 +0,0 @@
|
|||||||
# Multi-stage build for optimized Python application
|
|
||||||
FROM python:3.11-slim AS builder
|
|
||||||
|
|
||||||
# Set build-time environment variables
|
|
||||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
|
||||||
PYTHONUNBUFFERED=1 \
|
|
||||||
PIP_NO_CACHE_DIR=1 \
|
|
||||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
|
||||||
|
|
||||||
# Install build dependencies only
|
|
||||||
RUN apt-get update && apt-get install -y \
|
|
||||||
build-essential \
|
|
||||||
curl \
|
|
||||||
git \
|
|
||||||
&& rm -rf /var/lib/apt/lists/* && \
|
|
||||||
apt-get clean
|
|
||||||
|
|
||||||
# Configure SSH to trust GitHub host key
|
|
||||||
RUN mkdir -p ~/.ssh && \
|
|
||||||
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts && \
|
|
||||||
chmod 600 ~/.ssh/known_hosts
|
|
||||||
|
|
||||||
# Create virtual environment
|
|
||||||
RUN python -m venv /opt/venv
|
|
||||||
ENV PATH="/opt/venv/bin:$PATH"
|
|
||||||
|
|
||||||
# Upgrade pip and wheel for better caching
|
|
||||||
RUN pip install --upgrade pip setuptools wheel
|
|
||||||
|
|
||||||
# Copy requirements files for better Docker layer caching
|
|
||||||
COPY requirements.txt ./
|
|
||||||
|
|
||||||
# Install only production dependencies with no cache
|
|
||||||
RUN --mount=type=ssh echo "=== Installing dependencies ===" && \
|
|
||||||
pip install --no-cache-dir -r requirements.txt && \
|
|
||||||
echo "=== Dependencies installed successfully ===" && \
|
|
||||||
pip list | wc -l && \
|
|
||||||
echo "=== Cleaning cache files ===" && \
|
|
||||||
find /opt/venv -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true && \
|
|
||||||
find /opt/venv -name "*.pyc" -delete 2>/dev/null || true && \
|
|
||||||
rm -rf /root/.cache/pip/* && \
|
|
||||||
echo "=== Cleaning venv site-packages ===" && \
|
|
||||||
find /opt/venv/lib/python3.11/site-packages/ -type f -name "*.md" -delete 2>/dev/null || true && \
|
|
||||||
echo "=== Stripping .so files ===" && \
|
|
||||||
find /opt/venv -name "*.so" -exec strip {} + 2>/dev/null || true
|
|
||||||
|
|
||||||
# Production stage using python-slim for better functionality
|
|
||||||
FROM python:3.11-slim AS production
|
|
||||||
|
|
||||||
# Set environment variables
|
|
||||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
|
||||||
PYTHONUNBUFFERED=1 \
|
|
||||||
PATH="/opt/venv/bin:$PATH" \
|
|
||||||
POD_ID=unknown \
|
|
||||||
HOME="/app"
|
|
||||||
|
|
||||||
# Copy virtual environment from builder stage
|
|
||||||
COPY --from=builder /opt/venv /opt/venv
|
|
||||||
|
|
||||||
# Set working directory
|
|
||||||
WORKDIR /app
|
|
||||||
|
|
||||||
# Copy application code
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# Create necessary directories for runtime file creation
|
|
||||||
RUN mkdir -p /app/model_manager/reports /app/logs /app/temp /app/models /app/data && \
|
|
||||||
chmod 755 /app/model_manager/reports /app/logs /app/temp /app/models /app/data
|
|
||||||
|
|
||||||
# Create non-root user
|
|
||||||
RUN groupadd -r appuser && useradd -r -g appuser appuser && \
|
|
||||||
chown -R appuser:appuser /app
|
|
||||||
|
|
||||||
# Switch to non-root user
|
|
||||||
USER appuser
|
|
||||||
|
|
||||||
# Set entrypoint for proper signal handling and PID 1
|
|
||||||
ENTRYPOINT ["/opt/venv/bin/python", "-m", "model_manager.worker.worker"]
|
|
||||||
78
README.md
78
README.md
@@ -74,7 +74,7 @@ An enterprise-grade ML model training orchestration platform built on Temporal.
|
|||||||
- [Usage](#usage)
|
- [Usage](#usage)
|
||||||
- [Command Reference](#command-reference)
|
- [Command Reference](#command-reference)
|
||||||
- [Docker](#docker)
|
- [Docker](#docker)
|
||||||
- [Helm Chart](#helm-chart)
|
- [Kubernetes Deployment](#kubernetes-deployment)
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
@@ -1186,8 +1186,6 @@ sientia-dataops-model-manager/
|
|||||||
├── .github/workflows/ # CI/CD workflows
|
├── .github/workflows/ # CI/CD workflows
|
||||||
│ ├── quality-gate.yml # PR quality checks
|
│ ├── quality-gate.yml # PR quality checks
|
||||||
│ └── deploy.yml # Deployment workflow
|
│ └── deploy.yml # Deployment workflow
|
||||||
├── Dockerfile # Container image definition
|
|
||||||
├── values.yaml # Helm chart values
|
|
||||||
├── pyproject.toml # Project configuration
|
├── pyproject.toml # Project configuration
|
||||||
├── requirements.txt # Production dependencies
|
├── requirements.txt # Production dependencies
|
||||||
├── requirements-dev.txt # Development dependencies
|
├── requirements-dev.txt # Development dependencies
|
||||||
@@ -1475,79 +1473,17 @@ act -n
|
|||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
### Create image
|
Model Manager runs on the shared **`sientia-module`** image — the application code is cloned from the internal Gitea at runtime, not baked into an image. The image is defined in the [Aignosi/sientia-container-images](https://github.com/Aignosi/sientia-container-images) repository (`templates/sientia-module/`) and published to GCP Artifact Registry (`southamerica-east1-docker.pkg.dev/sientia-dev/sientia/sientia-module`).
|
||||||
|
|
||||||
```bash
|
The legacy repo-specific Dockerfile was removed — no image is built from this repo.
|
||||||
$ docker build --ssh default --no-cache --progress=plain -t aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0 .
|
|
||||||
```
|
|
||||||
|
|
||||||
### Create container
|
To run locally, use `./run_local.sh` (see [Local Execution](#local-execution)).
|
||||||
|
|
||||||
```bash
|
## Kubernetes Deployment
|
||||||
$ docker run --env-file .env --network="host" --name sientia-dataops-model-manager -d aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0
|
|
||||||
|
|
||||||
$ docker logs -f sientia-dataops-model-manager
|
Kubernetes deployment is managed by the [Aignosi/sientia-helm-chart](https://github.com/Aignosi/sientia-helm-chart) umbrella chart (`src/charts-internal/model-manager/`, which wraps the shared `sientia-module` chart) — the local `values.yaml` and the vendored `sientia-module/` chart copy were removed. Worker configuration, Gitea repo/branch, task queues, and sizing are all defined in the umbrella values.
|
||||||
```
|
|
||||||
|
|
||||||
### Login using access token
|
The standalone `sientia-module` chart still lives in [sientia-dataops-helm-repo](https://github.com/Aignosi/sientia-dataops-helm-repo) for the legacy deploy flow (temporarily broken until the CI/CD redesign).
|
||||||
|
|
||||||
```bash
|
|
||||||
$ docker login -u <username> -p <access-token> aignosi.azurecr.io
|
|
||||||
```
|
|
||||||
|
|
||||||
### Push image to repository
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ docker push aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0
|
|
||||||
```
|
|
||||||
|
|
||||||
## Helm Chart
|
|
||||||
|
|
||||||
### Reference
|
|
||||||
|
|
||||||
https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/274563074/Como+utilizar+o+Helm+Repo+Privado
|
|
||||||
|
|
||||||
### Add Helm Chart repository
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ helm repo add sientia \
|
|
||||||
https://raw.githubusercontent.com/Aignosi/sientia-dataops-helm-repo/refs/heads/main/ \
|
|
||||||
--username $GITHUB_USER \
|
|
||||||
--password $GITHUB_PASS
|
|
||||||
|
|
||||||
# Update repository
|
|
||||||
$ helm repo update
|
|
||||||
|
|
||||||
# List repositories
|
|
||||||
$ helm repo list
|
|
||||||
|
|
||||||
# List versions of a specific chart
|
|
||||||
$ helm search repo sientia --versions
|
|
||||||
|
|
||||||
# List all charts available
|
|
||||||
$ helm search repo sientia
|
|
||||||
|
|
||||||
# List chart details
|
|
||||||
$ helm show all sientia/sientia-module
|
|
||||||
|
|
||||||
# Download chart to current directory
|
|
||||||
$ helm pull sientia/sientia-module --version 0.6.0 --untar
|
|
||||||
|
|
||||||
# Remove chart directory
|
|
||||||
$ rm -rf sientia-module
|
|
||||||
```
|
|
||||||
|
|
||||||
### Helm Install
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ helm upgrade --install sientia-dataops-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.0
|
|
||||||
```
|
|
||||||
|
|
||||||
### Uninstall Helm Chart
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ helm uninstall sientia-dataops-model-manager -n sientia
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "model-manager"
|
name = "model-manager"
|
||||||
version = "1.2.0"
|
version = "2.0.0"
|
||||||
description = "Sientia DataOps Model Manager - ML Model Orchestration System"
|
description = "Sientia DataOps Model Manager - ML Model Orchestration System"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
sonar.projectKey=Aignosi_sientia-dataops-model-manager_dc6e051c-995a-4b57-9cde-f511018184c7
|
sonar.projectKey=Aignosi_sientia-dataops-model-manager_ac6a7d4d-508d-4cbf-bd3c-1bb0d79db881
|
||||||
sonar.projectName=sientia-dataops-model-manager
|
sonar.projectName=sientia-dataops-model-manager
|
||||||
sonar.sources=model_manager
|
sonar.sources=model_manager
|
||||||
sonar.tests=tests
|
sonar.tests=tests
|
||||||
|
|||||||
388
values.yaml
388
values.yaml
@@ -1,388 +0,0 @@
|
|||||||
#
|
|
||||||
# Default values for sientia-model-manager using the sientia-module chart (0.6.x).
|
|
||||||
# This is a YAML-formatted file.
|
|
||||||
# Declare variables to be passed into your templates.
|
|
||||||
#
|
|
||||||
|
|
||||||
projectName: &projectName "sientia-model-manager"
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Global configuration shared by all runtimes
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
global:
|
|
||||||
# Namespace used by the chart.
|
|
||||||
namespace: sientia
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Image configuration (chart-level)
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# The sientia-module chart allows overriding the image used by all runtimes.
|
|
||||||
# Per requirement, we deploy using the sientia-module image v1.0.0.
|
|
||||||
image:
|
|
||||||
repository: aignosi.azurecr.io/sientia-module
|
|
||||||
pullPolicy: Always
|
|
||||||
tag: "1.0.2"
|
|
||||||
|
|
||||||
# Common labels applied to pods (can be extended per project).
|
|
||||||
commonLabels: {}
|
|
||||||
|
|
||||||
# Resources inherited by all runtimes unless overridden.
|
|
||||||
resources: {}
|
|
||||||
# We usually recommend not to specify default resources and to leave this as a conscious
|
|
||||||
# choice for the user. This also increases chances charts run on environments with little
|
|
||||||
# resources, such as Minikube. If you do want to specify resources, uncomment the following
|
|
||||||
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
|
|
||||||
# limits:
|
|
||||||
# cpu: 100m
|
|
||||||
# memory: 128Mi
|
|
||||||
# requests:
|
|
||||||
# cpu: 100m
|
|
||||||
# memory: 128Mi
|
|
||||||
|
|
||||||
# Probes inherited by all runtimes unless overridden.
|
|
||||||
# More information:
|
|
||||||
# https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
|
|
||||||
livenessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python3
|
|
||||||
- -c
|
|
||||||
- "import requests; requests.get('http://localhost:9090/metrics')"
|
|
||||||
initialDelaySeconds: 20
|
|
||||||
periodSeconds: 30
|
|
||||||
|
|
||||||
readinessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python3
|
|
||||||
- -c
|
|
||||||
- "import requests; requests.get('http://localhost:9090/metrics')"
|
|
||||||
initialDelaySeconds: 10
|
|
||||||
periodSeconds: 15
|
|
||||||
|
|
||||||
# Autoscaling configuration inherited by all runtimes unless overridden.
|
|
||||||
# More information:
|
|
||||||
# https://kubernetes.io/docs/concepts/workloads/autoscaling/
|
|
||||||
autoscaling:
|
|
||||||
enabled: false
|
|
||||||
minReplicas: 1
|
|
||||||
maxReplicas: 100
|
|
||||||
targetCPUUtilizationPercentage: 80
|
|
||||||
# targetMemoryUtilizationPercentage: 80
|
|
||||||
|
|
||||||
# Environment variables shared by all runtimes.
|
|
||||||
env:
|
|
||||||
# Entrypoint variables
|
|
||||||
- name: GITHUB_REPO_URL
|
|
||||||
value: "git@github.com:Aignosi/sientia-dataops-model-manager.git"
|
|
||||||
- name: GITHUB_BRANCH
|
|
||||||
value: "release/SIENTIAPDE-1645"
|
|
||||||
- name: PYTHON_APP
|
|
||||||
value: "model_manager.worker.worker"
|
|
||||||
- name: PYPI_SERVER
|
|
||||||
value: "http://library-distribution-server.library.svc.cluster.local:5000"
|
|
||||||
|
|
||||||
- name: POSTGRES_HOST
|
|
||||||
value: "paradedb-rw.paradedb.svc.cluster.local"
|
|
||||||
- name: POSTGRES_PORT
|
|
||||||
value: "5432"
|
|
||||||
- name: POSTGRES_USER
|
|
||||||
value: "postgres"
|
|
||||||
- name: POSTGRES_PASSWORD
|
|
||||||
value: "nFqc81y6kwmr2zuAIx43DhiOosFCVPpeEfTtTWZflkNjB2j1KtEeIANkhFR9mAX3"
|
|
||||||
- name: POSTGRES_DBNAME
|
|
||||||
value: "sientia-core-mlops-bff"
|
|
||||||
- name: POSTGRES_MIN_CONNECTIONS
|
|
||||||
value: "10"
|
|
||||||
- name: POSTGRES_MAX_CONNECTIONS
|
|
||||||
value: "30"
|
|
||||||
|
|
||||||
- name: MLFLOW_URL
|
|
||||||
value: "http://sientia-tracker-mlflow-tracking.sientia-tracker.svc.cluster.local:80"
|
|
||||||
- name: MLFLOW_USERNAME
|
|
||||||
value: "aignosi"
|
|
||||||
- name: MLFLOW_PASSWORD
|
|
||||||
value: "1L0FP50j3ncp123"
|
|
||||||
|
|
||||||
- name: LOG_LEVEL
|
|
||||||
value: "DEBUG"
|
|
||||||
- name: HTTP_METRICS_PORT
|
|
||||||
value: "9090"
|
|
||||||
- name: HTTP_SDK_METRICS_PORT
|
|
||||||
value: "9091"
|
|
||||||
|
|
||||||
- name: TEMPORAL_HOST
|
|
||||||
value: "temporal-frontend.temporal.svc.cluster.local:7233"
|
|
||||||
- name: TEMPORAL_NAMESPACE
|
|
||||||
value: "model-manager"
|
|
||||||
- name: TRAIN_TASK_QUEUE
|
|
||||||
value: "train_model-queue"
|
|
||||||
- name: CLEANUP_TASK_QUEUE
|
|
||||||
value: "cleanup-queue"
|
|
||||||
- name: TEMPORAL_USE_TLS
|
|
||||||
value: "false"
|
|
||||||
|
|
||||||
- name: STORE_BASE_URL
|
|
||||||
value: "http://gitea-http.gitea.svc.cluster.local:3000"
|
|
||||||
- name: STORE_OWNER
|
|
||||||
value: "aignosi"
|
|
||||||
- name: STORE_REPO
|
|
||||||
value: "suse-model-store"
|
|
||||||
- name: STORE_USERNAME
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: sientia-plugin-store-credentials
|
|
||||||
key: username
|
|
||||||
- name: STORE_PASSWORD
|
|
||||||
valueFrom:
|
|
||||||
secretKeyRef:
|
|
||||||
name: sientia-plugin-store-credentials
|
|
||||||
key: password
|
|
||||||
- name: STORE_CACHE_TTL_SECONDS
|
|
||||||
value: "3600"
|
|
||||||
|
|
||||||
- name: MONGODB_USERNAME
|
|
||||||
value: "root"
|
|
||||||
- name: MONGODB_PASSWORD
|
|
||||||
value: "wKZDbMNU1c"
|
|
||||||
- name: MONGODB_URL
|
|
||||||
value: "my-release-mongodb.mongodb.svc.cluster.local:27017"
|
|
||||||
- name: MONGODB_DATABASE
|
|
||||||
value: "sientia"
|
|
||||||
- name: MONGODB_TTL_INDEX_HOURS
|
|
||||||
value: "1"
|
|
||||||
|
|
||||||
- name: MINIO_ENDPOINT_URL
|
|
||||||
value: "http://minio.minio.svc.cluster.local:9000"
|
|
||||||
- name: MINIO_ACCESS_KEY
|
|
||||||
value: "model-training-user"
|
|
||||||
- name: MINIO_SECRET_KEY
|
|
||||||
value: "modelTrainingUser123"
|
|
||||||
- name: MINIO_REGION
|
|
||||||
value: "us-east-1"
|
|
||||||
- name: MINIO_SECURE
|
|
||||||
value: "false"
|
|
||||||
- name: MINIO_MAX_RETRY_ATTEMPTS
|
|
||||||
value: "3"
|
|
||||||
- name: MINIO_RETRY_MODE
|
|
||||||
value: "adaptive"
|
|
||||||
- name: MINIO_CONNECT_TIMEOUT
|
|
||||||
value: "10"
|
|
||||||
- name: MINIO_READ_TIMEOUT
|
|
||||||
value: "60"
|
|
||||||
- name: MINIO_DEFAULT_BUCKET
|
|
||||||
value: "model-training"
|
|
||||||
|
|
||||||
- name: TIMEOUT_VALIDATE_PARAMS
|
|
||||||
value: "30"
|
|
||||||
- name: TIMEOUT_TRAIN_MODEL
|
|
||||||
value: "2700"
|
|
||||||
- name: TIMEOUT_DELETE_FILE
|
|
||||||
value: "120"
|
|
||||||
- name: TIMEOUT_UPDATE_DATABASE
|
|
||||||
value: "30"
|
|
||||||
|
|
||||||
- name: CLEANUP_RETENTION_HOURS
|
|
||||||
value: "24"
|
|
||||||
- name: CLEANUP_DRY_RUN
|
|
||||||
value: "false"
|
|
||||||
- name: TIMEOUT_CLEANUP_MINIO
|
|
||||||
value: "300"
|
|
||||||
- name: TIMEOUT_CLEANUP_LOCAL
|
|
||||||
value: "120"
|
|
||||||
- name: MAX_KEYS_CLEANUP
|
|
||||||
value: "1000"
|
|
||||||
- name: DEFAULT_CLEANUP_BUCKET
|
|
||||||
value: "model-training"
|
|
||||||
|
|
||||||
# Cleanup Schedule Configuration
|
|
||||||
- name: CLEANUP_SCHEDULE_ID
|
|
||||||
value: "cleanup-files-daily"
|
|
||||||
- name: CLEANUP_CRON
|
|
||||||
value: "0 0 * * *" # Midnight UTC
|
|
||||||
- name: CLEANUP_TIMEZONE
|
|
||||||
value: "UTC"
|
|
||||||
- name: CLEANUP_EXECUTION_TIMEOUT_HOURS
|
|
||||||
value: "1"
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Runtimes configuration
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Each runtime inherits settings from `global` (resources, env, probes, autoscaling)
|
|
||||||
# unless overridden here.
|
|
||||||
runtimes:
|
|
||||||
basic:
|
|
||||||
# Replicas for this runtime. Replaces the old replicaCount.
|
|
||||||
replicas: 1
|
|
||||||
xgboost:
|
|
||||||
replicas: 1
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Chart-level configuration (applies to all runtimes)
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
|
|
||||||
# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
|
|
||||||
imagePullSecrets:
|
|
||||||
- name: docker-hub-secret
|
|
||||||
|
|
||||||
# This is to override the chart name.
|
|
||||||
nameOverride: *projectName
|
|
||||||
fullnameOverride: *projectName
|
|
||||||
|
|
||||||
# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
|
|
||||||
serviceAccount:
|
|
||||||
# Specifies whether a service account should be created
|
|
||||||
create: true
|
|
||||||
# Automatically mount a ServiceAccount's API credentials?
|
|
||||||
automount: true
|
|
||||||
# Annotations to add to the service account
|
|
||||||
annotations: {}
|
|
||||||
# The name of the service account to use.
|
|
||||||
# If not set and create is true, a name is generated using the fullname template
|
|
||||||
name: *projectName
|
|
||||||
|
|
||||||
# This is for setting Kubernetes Annotations to a Pod.
|
|
||||||
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
|
|
||||||
podAnnotations: {}
|
|
||||||
# This is for setting Kubernetes Labels to a Pod.
|
|
||||||
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
|
||||||
podLabels: {}
|
|
||||||
|
|
||||||
podSecurityContext: {}
|
|
||||||
# fsGroup: 2000
|
|
||||||
|
|
||||||
securityContext: {}
|
|
||||||
# capabilities:
|
|
||||||
# drop:
|
|
||||||
# - ALL
|
|
||||||
# readOnlyRootFilesystem: true
|
|
||||||
# runAsNonRoot: true
|
|
||||||
# runAsUser: 1000
|
|
||||||
|
|
||||||
# Additional volumes on the output Deployment definition.
|
|
||||||
volumes:
|
|
||||||
- name: model-manager-runtime
|
|
||||||
emptyDir:
|
|
||||||
sizeLimit: 1Gi
|
|
||||||
|
|
||||||
# Additional volumeMounts on the output Deployment definition.
|
|
||||||
volumeMounts:
|
|
||||||
- name: model-manager-runtime
|
|
||||||
mountPath: "/var/lib/model-manager"
|
|
||||||
|
|
||||||
# Deployment strategy configuration
|
|
||||||
# More information: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy
|
|
||||||
deploymentStrategy:
|
|
||||||
type: Recreate
|
|
||||||
# rollingUpdate:
|
|
||||||
# maxSurge: 0
|
|
||||||
# maxUnavailable: 1
|
|
||||||
|
|
||||||
# Number of old ReplicaSets to retain
|
|
||||||
revisionHistoryLimit: 2
|
|
||||||
|
|
||||||
nodeSelector: {}
|
|
||||||
|
|
||||||
tolerations: []
|
|
||||||
|
|
||||||
affinity: {}
|
|
||||||
|
|
||||||
services:
|
|
||||||
sdk-metrics:
|
|
||||||
enabled: true
|
|
||||||
type: ClusterIP
|
|
||||||
port: 9091
|
|
||||||
targetPort: 9091
|
|
||||||
name: sdk-metrics
|
|
||||||
metrics:
|
|
||||||
enabled: true
|
|
||||||
type: ClusterIP
|
|
||||||
port: 9090
|
|
||||||
targetPort: 9090
|
|
||||||
name: metrics
|
|
||||||
|
|
||||||
# Configuração do ServiceMonitor para o Prometheus Operator
|
|
||||||
# ref: https://github.com/prometheus-operator/prometheus-operator
|
|
||||||
serviceMonitor:
|
|
||||||
# Se true, um recurso ServiceMonitor será criado.
|
|
||||||
enabled: true
|
|
||||||
# O intervalo no qual as métricas devem ser coletadas (ex: 30s, 1m).
|
|
||||||
endpoints:
|
|
||||||
- port: metrics
|
|
||||||
path: /metrics
|
|
||||||
interval: 30s
|
|
||||||
relabelings: []
|
|
||||||
- port: sdk-metrics
|
|
||||||
path: /metrics
|
|
||||||
interval: 30s
|
|
||||||
relabelings: []
|
|
||||||
additionalLabels:
|
|
||||||
release: kube-prometheus-stack
|
|
||||||
|
|
||||||
ssh:
|
|
||||||
enabled: true
|
|
||||||
secretName: git-ssh-key-sientia-model-manager-worker
|
|
||||||
sshPath: /mnt/.ssh
|
|
||||||
knownHostsPath: /mnt/known_hosts
|
|
||||||
|
|
||||||
# Configuração para dashboards do Grafana
|
|
||||||
grafanaDashboard:
|
|
||||||
# Habilita a criação de ConfigMaps para dashboards
|
|
||||||
enabled: true
|
|
||||||
# Namespace onde o Grafana está instalado (ajuste conforme seu ambiente)
|
|
||||||
namespace: monitoring
|
|
||||||
# Labels para que o sidecar do Grafana encontre os dashboards
|
|
||||||
labels:
|
|
||||||
grafana_dashboard: "1"
|
|
||||||
# Lista de dashboards para importar
|
|
||||||
dashboards:
|
|
||||||
- name: sientia-dataops-model-manager
|
|
||||||
title: "Sientia DataOps Model Manager"
|
|
||||||
uid: "sientia-dataops-model-manager"
|
|
||||||
folder: "Sientia"
|
|
||||||
jsonFile: "dashboards/sientia-dataops-model-manager.json"
|
|
||||||
overwrite: true # Sobrescreve dashboard se já existir
|
|
||||||
version: "1.0.0" # Version inicial do dashboard
|
|
||||||
|
|
||||||
# Configuração para datasources do Grafana
|
|
||||||
grafanaDatasource:
|
|
||||||
# Habilita a criação de ConfigMap para datasources
|
|
||||||
enabled: false
|
|
||||||
# Namespace onde o Grafana está instalado
|
|
||||||
namespace: monitoring
|
|
||||||
# Labels para que o sidecar do Grafana encontre os datasources
|
|
||||||
labels:
|
|
||||||
grafana_datasource: "1"
|
|
||||||
# Lista de datasources para configurar
|
|
||||||
datasources: []
|
|
||||||
# Exemplo de datasource:
|
|
||||||
# - name: Prometheus
|
|
||||||
# type: prometheus
|
|
||||||
# url: http://prometheus-server.monitoring.svc.cluster.local
|
|
||||||
# isDefault: true
|
|
||||||
# jsonData:
|
|
||||||
# timeInterval: "5s"
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# Helm usage examples
|
|
||||||
# -----------------------------------------------------------------------------
|
|
||||||
# kubectl create secret docker-registry docker-hub-secret --namespace sientia --docker-server=http://aignosi.azurecr.io --docker-username=aignosi --docker-password=<pwd>
|
|
||||||
#
|
|
||||||
# helm upgrade --install sientia-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.1
|
|
||||||
#
|
|
||||||
# Global/runtimes layout note:
|
|
||||||
# - Shared configuration lives under `global` (env, probes, autoscaling, namespace).
|
|
||||||
# - Individual runtimes are defined under `runtimes`, each with its own `name` and `replicas`.
|
|
||||||
# - Runtimes inherit `global` settings unless overridden at the runtime level.
|
|
||||||
#
|
|
||||||
# kubectl create secret generic git-ssh-key-sientia-model-manager-worker \
|
|
||||||
# --namespace sientia \
|
|
||||||
# --from-file=ssh-privatekey=git_key \
|
|
||||||
# --type=kubernetes.io/ssh-auth
|
|
||||||
|
|
||||||
# kubectl create secret generic sientia-plugin-store-credentials \
|
|
||||||
# --namespace sientia \
|
|
||||||
# --from-literal=username=<username> \
|
|
||||||
# --from-literal=password=<password>
|
|
||||||
|
|
||||||
|
|
||||||
Reference in New Issue
Block a user