SIENTIAPDE-1645: Overhaul deployment, add Claude AI guidance, and enhance CI/CD.

Migrated to a shared sientia-module image and umbrella Helm chart, removing the standalone Dockerfile, .dockerignore, and values.yaml. Introduced CLAUDE.md for AI-assisted development. Updated quality-gate.yml with push triggers, concurrency, and refined permissions. Bumped project version to 2.0.0 to reflect these architectural changes.
This commit is contained in:
Bruno Domingues
2026-08-05 10:38:38 -03:00
8 changed files with 100 additions and 633 deletions

View File

@@ -1,93 +0,0 @@
# ============================================================================
# WHITELIST APPROACH: Block everything by default, then allow only what's needed
# ============================================================================
# Block everything first
*
# ============================================================================
# ALLOW: Application source code (model_manager package)
# ============================================================================
# Allow the main package directory and all Python files
!model_manager/
!model_manager/**/*.py
!model_manager/**/__init__.py
# Allow subdirectories structure
!model_manager/activities/
!model_manager/activities/**
!model_manager/schedules/
!model_manager/schedules/**
!model_manager/sientia/
!model_manager/sientia/**
!model_manager/utils/
!model_manager/utils/**
!model_manager/utils/models/
!model_manager/utils/models/**
!model_manager/utils/repository/
!model_manager/utils/repository/**
!model_manager/worker/
!model_manager/worker/**
!model_manager/workflows/
!model_manager/workflows/**
# Allow reports directory with header.html
!model_manager/reports/
!model_manager/reports/header.html
# Allow temp directory structure (but not its contents)
!model_manager/reports/temp/
# ============================================================================
# ALLOW: Dependencies file (needed for pip install in Dockerfile)
# ============================================================================
!requirements.txt
# ============================================================================
# BLOCK: Explicitly block unwanted files even if they match above patterns
# ============================================================================
# Python cache and compiled files
**/__pycache__/
**/*.pyc
**/*.pyo
**/*.pyd
**/.Python
**/*.so
**/*.egg
**/*.egg-info/
# Tests (not needed in production)
model_manager/**/test_*.py
model_manager/**/*_test.py
# IDE and editor files
**/.vscode/
**/.idea/
**/*.swp
**/*.swo
**/*~
# OS files
**/.DS_Store
**/Thumbs.db
# Logs and temporary files
**/*.log
**/*.tmp
**/*.temp
# Local configuration
**/.env
**/.env.local
**/*.local
# Documentation inside code
**/*.md
**/README*
# Backup files
**/*.bak
**/*.backup
**/*.old

View File

@@ -1,6 +1,11 @@
name: Quality gate name: Quality gate
on: on:
push:
branches:
- main
- 'release/**'
- 'feature/**'
pull_request: pull_request:
branches: branches:
- main - main
@@ -8,10 +13,17 @@ on:
- 'feature/**' - 'feature/**'
types: [ opened, synchronize, reopened ] types: [ opened, synchronize, reopened ]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs: jobs:
quality-gate: quality-gate:
uses: Aignosi/github_workflow_templates/.github/workflows/python-quality-gate.yml@main uses: Aignosi/github_workflow_templates/.github/workflows/python-quality-gate.yml@main
permissions: write-all permissions:
contents: read
pull-requests: write
issues: write
with: with:
project_name: 'model_manager' project_name: 'model_manager'
repositories: 'sientia-dataops-library,sientia-model-library' repositories: 'sientia-dataops-library,sientia-model-library'

78
CLAUDE.md Normal file
View File

@@ -0,0 +1,78 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
Enterprise ML model training orchestration platform built on **Temporal**. Orchestrates training workflows (validation → train → save to MLFlow → cleanup), backed by PostgreSQL (experiment tracking), MinIO (data/artifacts), MongoDB (notifications), and a Git-based "Plugin Store" that supplies model runtimes/wrappers dynamically. Python 3.11+.
## Commands
```bash
# Local run (loads .env, starts worker)
./run_local.sh
# or manually:
python -m model_manager.worker.worker
# Full validation (format check, lint, mypy, bandit, tests w/ coverage) — run before committing
./validate.sh
./validate.sh --fix # auto-fix format/lint, no tests
./validate.sh --no-tests # skip tests
./validate.sh --only-tests # tests only
# Individual tools
ruff format --check model_manager/ tests/
ruff format model_manager/ tests/ # auto-fix
ruff check model_manager/ tests/
ruff check --fix model_manager/ tests/
mypy model_manager/
bandit -r model_manager/ -ll
# Tests
pytest # runs tests/ and e2e/ (see pyproject testpaths)
pytest tests/ # unit tests only
pytest tests/activities/test_training.py -v # single file
pytest tests/activities/test_training.py::test_name -v # single test
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=html
# Manual integration/E2E scripts (require live Temporal/Postgres/MinIO stack; run via IDE "Run Cell" or directly)
python scripts/run_training_test.py # supports --all for batch scenario execution
python scripts/run_cleanup_test.py
```
Coverage gate in CI is 80% minimum; this repo targets 100% in practice — don't add uncovered code paths without tests.
## Architecture
### Temporal layering
- **Worker** (`model_manager/worker/worker.py`): entrypoint. Builds config from env, connects Postgres/MLFlow/MinIO/Plugin Store, installs the `RUNTIME` model runtime via `PluginStore.install_runtime`, starts two Temporal workers (one per task queue) plus Prometheus metrics server and the cleanup schedule. All heavy imports are wrapped in `workflow.unsafe.imports_passed_through()` — required because Temporal workflow sandboxing forbids non-deterministic imports at module scope outside that block.
- **Workflows** (`model_manager/workflows/`): pure orchestration, no I/O beyond `execute_activity_method` calls.
- `train_model.py` (`TrainModel`, workflow name `train_model`): validate `experiment_run_id``load_model_metadata``validate_train_params` → update DB status `ORCHESTRATOR_WAITING_PROC``train_model` activity → update DB status `TRAINING_SUCCESS`/`TRAINING_ERROR``cleanup_resources` (always runs in `finally` if a `run_dir` exists, even on failure). Uses distinct `RetryPolicy`s per operation class: `network_retry_policy` (transient I/O), `no_retry_policy` (training/validation — data errors are permanent, and retrying would duplicate MLflow writes), `database_retry_policy` (Postgres). Timeouts are env-driven (`TIMEOUT_*`), extend `TIMEOUT_TRAIN_MODEL` instead of adding retries for slow training.
- `cleanup_files.py` (`CleanupFiles`): scheduled (`cleanup_schedule.py`, default daily at midnight UTC via `CLEANUP_CRON`) deletion of stale local temp dirs matching `name_YYYYMMDD_HHMMSS_microseconds`, older than `CLEANUP_RETENTION_HOURS`.
- Task queues are derived from `RUNTIME` env var (default `single`) via `prepare_worker.build_queue_name`: `train_model-<runtime>-queue`, `cleanup_files-<runtime>-queue`. A Temporal client starting a workflow must target the matching queue.
- **Activities** (`model_manager/activities/`): `Activities` (`activities.py`) combines `ExperimentTracking` + `Training` + `Cleanup` via multiple inheritance into one class registered with the worker — mind MRO and `__init__`/`__del__` ordering when touching this class (see the defensive `hasattr(self, 'engine')` check in `__del__`).
- `experiment_tracking.py`: single `update_experiment_run()` entrypoint handling three update kinds via `UpdateType`: `STATUS`, `STATUS_WITH_ERROR`, `MODEL_SAVED`. Error messages truncated to 1024 chars before persisting.
- `training.py`: single `train_model()` entrypoint. Receives pre-downloaded data as `BytesIO` (not re-downloaded per-activity) to avoid memory leaks; never raises on training failure — returns a result/error so the workflow can update DB status without an unhandled exception mid-activity.
- `cleanup.py`: local-filesystem-only cleanup (`cleanup_temp_directories`), no MinIO involved — MinIO artifacts are managed externally.
### Data/config layer
- `model_manager/utils/connectors_config.py`: single source of truth for env-var → config-dict builders (`build_postgres_config`, `build_mlflow_config`, `build_minio_config`, `build_mongodb_config`, `build_plugin_store_config`). Read this file (and `worker.py`) before trusting any env var default — the README table mirrors it but this file is authoritative.
- `model_manager/utils/models/train_model_params.py`: `TrainModelParams` — the workflow's single input contract. `from_dict` does type coercion; `validate_business_rules()` runs after, enforcing 7 rules (train_size range, non-empty variable_columns, required model_metadata, JSON-Schema validation of `data_model_kwargs`/`model_kwargs`/`opt_params` against schemas in `model_metadata` via `Draft202012Validator`, required non-blank strings, allowed `date_format` values, numeric `experiment_run_id`). Extending training params means updating this file's validation, not just the workflow.
- `model_manager/utils/repository/data_manager_repository.py`: data loading, feature prep, metrics, report generation — the actual ML/data logic activities delegate into.
- `model_manager/sientia/`: RCE drift metrics (`metrics.py`), Evidently-based HTML report generation (`reports.py`), custom exceptions.
- `model_manager/metrics.py`: Prometheus metric definitions (`app_up`, `workflow_execution_total`, `activity_execution_total`, training/cleanup/RCE-drift gauges). New metrics go here, not inline in activities.
### External dependencies
Two private Aignosi packages carry non-trivial logic: `sientia_do` (observability/logger, notification handler, metrics controller, Postgres/MinIO repositories) and `sientia_model` (Plugin Store, MLflow repository, model wrappers). Both are pinned in `requirements.txt`/`requirements-local.txt`; when behavior looks like it lives outside this repo, it's likely in one of these.
### Test bootstrap quirk
`tests/conftest.py` stubs `evidently` and parts of `sientia_do` at `pytest_configure` time so unit tests can import `model_manager.sientia.reports` without the full (heavy/optional) Evidently install. If you add new imports from these packages in production code paths that unit tests exercise, you likely need to extend the stubs here too.
### Test layout
- `tests/` — unit tests, mirrors `model_manager/` package structure (`tests/activities/`, `tests/schedules/`, `tests/sientia/`, etc.).
- `e2e/` — integration tests against a real Temporal/Postgres/MinIO/MongoDB stack (via `testcontainers`), driven by JSON scenario files in `docs/test-scenarios/` (see `e2e/scenarios.md` for the scenario→test mapping). Scenario payloads are snake_case, shaped like `TrainModelParams`/`input-sample.json`.
- New scenario: copy an existing `docs/test-scenarios/*.json`, add/extend a test in `e2e/test_train_model_workflow.py`, update `e2e/scenarios.md`.
## Conventions
- Ruff: line-length 100, single quotes, `select = [E,W,F,I,B,C4,UP,N,YTT,S,BLE,A,C90]`, max complexity 15. `N802`/`N806` ignored (Temporal decorators use non-lowercase names). `S101`/`S105`/`S106` ignored in tests.
- mypy: `disallow_untyped_defs = false` but `check_untyped_defs = true` — existing untyped defs are tolerated, but code inside them is still checked. Several external packages (`temporalio`, `mlflow`, `sientia_model`, etc.) have `ignore_missing_imports`; `model_repository.py`/`data_manager_repository.py` have `ignore_errors = true` (legacy — don't extend that list without reason).

View File

@@ -1,78 +0,0 @@
# Multi-stage build for optimized Python application
FROM python:3.11-slim AS builder
# Set build-time environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
# Install build dependencies only
RUN apt-get update && apt-get install -y \
build-essential \
curl \
git \
&& rm -rf /var/lib/apt/lists/* && \
apt-get clean
# Configure SSH to trust GitHub host key
RUN mkdir -p ~/.ssh && \
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts && \
chmod 600 ~/.ssh/known_hosts
# Create virtual environment
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
# Upgrade pip and wheel for better caching
RUN pip install --upgrade pip setuptools wheel
# Copy requirements files for better Docker layer caching
COPY requirements.txt ./
# Install only production dependencies with no cache
RUN --mount=type=ssh echo "=== Installing dependencies ===" && \
pip install --no-cache-dir -r requirements.txt && \
echo "=== Dependencies installed successfully ===" && \
pip list | wc -l && \
echo "=== Cleaning cache files ===" && \
find /opt/venv -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true && \
find /opt/venv -name "*.pyc" -delete 2>/dev/null || true && \
rm -rf /root/.cache/pip/* && \
echo "=== Cleaning venv site-packages ===" && \
find /opt/venv/lib/python3.11/site-packages/ -type f -name "*.md" -delete 2>/dev/null || true && \
echo "=== Stripping .so files ===" && \
find /opt/venv -name "*.so" -exec strip {} + 2>/dev/null || true
# Production stage using python-slim for better functionality
FROM python:3.11-slim AS production
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PATH="/opt/venv/bin:$PATH" \
POD_ID=unknown \
HOME="/app"
# Copy virtual environment from builder stage
COPY --from=builder /opt/venv /opt/venv
# Set working directory
WORKDIR /app
# Copy application code
COPY . .
# Create necessary directories for runtime file creation
RUN mkdir -p /app/model_manager/reports /app/logs /app/temp /app/models /app/data && \
chmod 755 /app/model_manager/reports /app/logs /app/temp /app/models /app/data
# Create non-root user
RUN groupadd -r appuser && useradd -r -g appuser appuser && \
chown -R appuser:appuser /app
# Switch to non-root user
USER appuser
# Set entrypoint for proper signal handling and PID 1
ENTRYPOINT ["/opt/venv/bin/python", "-m", "model_manager.worker.worker"]

View File

@@ -74,7 +74,7 @@ An enterprise-grade ML model training orchestration platform built on Temporal.
- [Usage](#usage) - [Usage](#usage)
- [Command Reference](#command-reference) - [Command Reference](#command-reference)
- [Docker](#docker) - [Docker](#docker)
- [Helm Chart](#helm-chart) - [Kubernetes Deployment](#kubernetes-deployment)
## Features ## Features
@@ -1186,8 +1186,6 @@ sientia-dataops-model-manager/
├── .github/workflows/ # CI/CD workflows ├── .github/workflows/ # CI/CD workflows
│ ├── quality-gate.yml # PR quality checks │ ├── quality-gate.yml # PR quality checks
│ └── deploy.yml # Deployment workflow │ └── deploy.yml # Deployment workflow
├── Dockerfile # Container image definition
├── values.yaml # Helm chart values
├── pyproject.toml # Project configuration ├── pyproject.toml # Project configuration
├── requirements.txt # Production dependencies ├── requirements.txt # Production dependencies
├── requirements-dev.txt # Development dependencies ├── requirements-dev.txt # Development dependencies
@@ -1475,79 +1473,17 @@ act -n
## Docker ## Docker
### Create image Model Manager runs on the shared **`sientia-module`** image — the application code is cloned from the internal Gitea at runtime, not baked into an image. The image is defined in the [Aignosi/sientia-container-images](https://github.com/Aignosi/sientia-container-images) repository (`templates/sientia-module/`) and published to GCP Artifact Registry (`southamerica-east1-docker.pkg.dev/sientia-dev/sientia/sientia-module`).
```bash The legacy repo-specific Dockerfile was removed — no image is built from this repo.
$ docker build --ssh default --no-cache --progress=plain -t aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0 .
```
### Create container To run locally, use `./run_local.sh` (see [Local Execution](#local-execution)).
```bash ## Kubernetes Deployment
$ docker run --env-file .env --network="host" --name sientia-dataops-model-manager -d aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0
$ docker logs -f sientia-dataops-model-manager Kubernetes deployment is managed by the [Aignosi/sientia-helm-chart](https://github.com/Aignosi/sientia-helm-chart) umbrella chart (`src/charts-internal/model-manager/`, which wraps the shared `sientia-module` chart) — the local `values.yaml` and the vendored `sientia-module/` chart copy were removed. Worker configuration, Gitea repo/branch, task queues, and sizing are all defined in the umbrella values.
```
### Login using access token The standalone `sientia-module` chart still lives in [sientia-dataops-helm-repo](https://github.com/Aignosi/sientia-dataops-helm-repo) for the legacy deploy flow (temporarily broken until the CI/CD redesign).
```bash
$ docker login -u <username> -p <access-token> aignosi.azurecr.io
```
### Push image to repository
```bash
$ docker push aignosi.azurecr.io/sientia-dataops-model-manager:0.0.0
```
## Helm Chart
### Reference
https://aignosi-wiki.atlassian.net/wiki/spaces/IT1/pages/274563074/Como+utilizar+o+Helm+Repo+Privado
### Add Helm Chart repository
```bash
$ helm repo add sientia \
https://raw.githubusercontent.com/Aignosi/sientia-dataops-helm-repo/refs/heads/main/ \
--username $GITHUB_USER \
--password $GITHUB_PASS
# Update repository
$ helm repo update
# List repositories
$ helm repo list
# List versions of a specific chart
$ helm search repo sientia --versions
# List all charts available
$ helm search repo sientia
# List chart details
$ helm show all sientia/sientia-module
# Download chart to current directory
$ helm pull sientia/sientia-module --version 0.6.0 --untar
# Remove chart directory
$ rm -rf sientia-module
```
### Helm Install
```shell
$ helm upgrade --install sientia-dataops-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.0
```
### Uninstall Helm Chart
```shell
$ helm uninstall sientia-dataops-model-manager -n sientia
```
--- ---

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "model-manager" name = "model-manager"
version = "1.2.0" version = "2.0.0"
description = "Sientia DataOps Model Manager - ML Model Orchestration System" description = "Sientia DataOps Model Manager - ML Model Orchestration System"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"

View File

@@ -1,4 +1,4 @@
sonar.projectKey=Aignosi_sientia-dataops-model-manager_dc6e051c-995a-4b57-9cde-f511018184c7 sonar.projectKey=Aignosi_sientia-dataops-model-manager_ac6a7d4d-508d-4cbf-bd3c-1bb0d79db881
sonar.projectName=sientia-dataops-model-manager sonar.projectName=sientia-dataops-model-manager
sonar.sources=model_manager sonar.sources=model_manager
sonar.tests=tests sonar.tests=tests

View File

@@ -1,388 +0,0 @@
#
# Default values for sientia-model-manager using the sientia-module chart (0.6.x).
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
#
projectName: &projectName "sientia-model-manager"
# -----------------------------------------------------------------------------
# Global configuration shared by all runtimes
# -----------------------------------------------------------------------------
global:
# Namespace used by the chart.
namespace: sientia
# -----------------------------------------------------------------------------
# Image configuration (chart-level)
# -----------------------------------------------------------------------------
# The sientia-module chart allows overriding the image used by all runtimes.
# Per requirement, we deploy using the sientia-module image v1.0.0.
image:
repository: aignosi.azurecr.io/sientia-module
pullPolicy: Always
tag: "1.0.2"
# Common labels applied to pods (can be extended per project).
commonLabels: {}
# Resources inherited by all runtimes unless overridden.
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
# Probes inherited by all runtimes unless overridden.
# More information:
# https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
livenessProbe:
exec:
command:
- python3
- -c
- "import requests; requests.get('http://localhost:9090/metrics')"
initialDelaySeconds: 20
periodSeconds: 30
readinessProbe:
exec:
command:
- python3
- -c
- "import requests; requests.get('http://localhost:9090/metrics')"
initialDelaySeconds: 10
periodSeconds: 15
# Autoscaling configuration inherited by all runtimes unless overridden.
# More information:
# https://kubernetes.io/docs/concepts/workloads/autoscaling/
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
# Environment variables shared by all runtimes.
env:
# Entrypoint variables
- name: GITHUB_REPO_URL
value: "git@github.com:Aignosi/sientia-dataops-model-manager.git"
- name: GITHUB_BRANCH
value: "release/SIENTIAPDE-1645"
- name: PYTHON_APP
value: "model_manager.worker.worker"
- name: PYPI_SERVER
value: "http://library-distribution-server.library.svc.cluster.local:5000"
- name: POSTGRES_HOST
value: "paradedb-rw.paradedb.svc.cluster.local"
- name: POSTGRES_PORT
value: "5432"
- name: POSTGRES_USER
value: "postgres"
- name: POSTGRES_PASSWORD
value: "nFqc81y6kwmr2zuAIx43DhiOosFCVPpeEfTtTWZflkNjB2j1KtEeIANkhFR9mAX3"
- name: POSTGRES_DBNAME
value: "sientia-core-mlops-bff"
- name: POSTGRES_MIN_CONNECTIONS
value: "10"
- name: POSTGRES_MAX_CONNECTIONS
value: "30"
- name: MLFLOW_URL
value: "http://sientia-tracker-mlflow-tracking.sientia-tracker.svc.cluster.local:80"
- name: MLFLOW_USERNAME
value: "aignosi"
- name: MLFLOW_PASSWORD
value: "1L0FP50j3ncp123"
- name: LOG_LEVEL
value: "DEBUG"
- name: HTTP_METRICS_PORT
value: "9090"
- name: HTTP_SDK_METRICS_PORT
value: "9091"
- name: TEMPORAL_HOST
value: "temporal-frontend.temporal.svc.cluster.local:7233"
- name: TEMPORAL_NAMESPACE
value: "model-manager"
- name: TRAIN_TASK_QUEUE
value: "train_model-queue"
- name: CLEANUP_TASK_QUEUE
value: "cleanup-queue"
- name: TEMPORAL_USE_TLS
value: "false"
- name: STORE_BASE_URL
value: "http://gitea-http.gitea.svc.cluster.local:3000"
- name: STORE_OWNER
value: "aignosi"
- name: STORE_REPO
value: "suse-model-store"
- name: STORE_USERNAME
valueFrom:
secretKeyRef:
name: sientia-plugin-store-credentials
key: username
- name: STORE_PASSWORD
valueFrom:
secretKeyRef:
name: sientia-plugin-store-credentials
key: password
- name: STORE_CACHE_TTL_SECONDS
value: "3600"
- name: MONGODB_USERNAME
value: "root"
- name: MONGODB_PASSWORD
value: "wKZDbMNU1c"
- name: MONGODB_URL
value: "my-release-mongodb.mongodb.svc.cluster.local:27017"
- name: MONGODB_DATABASE
value: "sientia"
- name: MONGODB_TTL_INDEX_HOURS
value: "1"
- name: MINIO_ENDPOINT_URL
value: "http://minio.minio.svc.cluster.local:9000"
- name: MINIO_ACCESS_KEY
value: "model-training-user"
- name: MINIO_SECRET_KEY
value: "modelTrainingUser123"
- name: MINIO_REGION
value: "us-east-1"
- name: MINIO_SECURE
value: "false"
- name: MINIO_MAX_RETRY_ATTEMPTS
value: "3"
- name: MINIO_RETRY_MODE
value: "adaptive"
- name: MINIO_CONNECT_TIMEOUT
value: "10"
- name: MINIO_READ_TIMEOUT
value: "60"
- name: MINIO_DEFAULT_BUCKET
value: "model-training"
- name: TIMEOUT_VALIDATE_PARAMS
value: "30"
- name: TIMEOUT_TRAIN_MODEL
value: "2700"
- name: TIMEOUT_DELETE_FILE
value: "120"
- name: TIMEOUT_UPDATE_DATABASE
value: "30"
- name: CLEANUP_RETENTION_HOURS
value: "24"
- name: CLEANUP_DRY_RUN
value: "false"
- name: TIMEOUT_CLEANUP_MINIO
value: "300"
- name: TIMEOUT_CLEANUP_LOCAL
value: "120"
- name: MAX_KEYS_CLEANUP
value: "1000"
- name: DEFAULT_CLEANUP_BUCKET
value: "model-training"
# Cleanup Schedule Configuration
- name: CLEANUP_SCHEDULE_ID
value: "cleanup-files-daily"
- name: CLEANUP_CRON
value: "0 0 * * *" # Midnight UTC
- name: CLEANUP_TIMEZONE
value: "UTC"
- name: CLEANUP_EXECUTION_TIMEOUT_HOURS
value: "1"
# -----------------------------------------------------------------------------
# Runtimes configuration
# -----------------------------------------------------------------------------
# Each runtime inherits settings from `global` (resources, env, probes, autoscaling)
# unless overridden here.
runtimes:
basic:
# Replicas for this runtime. Replaces the old replicaCount.
replicas: 1
xgboost:
replicas: 1
# -----------------------------------------------------------------------------
# Chart-level configuration (applies to all runtimes)
# -----------------------------------------------------------------------------
# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
imagePullSecrets:
- name: docker-hub-secret
# This is to override the chart name.
nameOverride: *projectName
fullnameOverride: *projectName
# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
serviceAccount:
# Specifies whether a service account should be created
create: true
# Automatically mount a ServiceAccount's API credentials?
automount: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: *projectName
# This is for setting Kubernetes Annotations to a Pod.
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
podAnnotations: {}
# This is for setting Kubernetes Labels to a Pod.
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
podLabels: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
# Additional volumes on the output Deployment definition.
volumes:
- name: model-manager-runtime
emptyDir:
sizeLimit: 1Gi
# Additional volumeMounts on the output Deployment definition.
volumeMounts:
- name: model-manager-runtime
mountPath: "/var/lib/model-manager"
# Deployment strategy configuration
# More information: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy
deploymentStrategy:
type: Recreate
# rollingUpdate:
# maxSurge: 0
# maxUnavailable: 1
# Number of old ReplicaSets to retain
revisionHistoryLimit: 2
nodeSelector: {}
tolerations: []
affinity: {}
services:
sdk-metrics:
enabled: true
type: ClusterIP
port: 9091
targetPort: 9091
name: sdk-metrics
metrics:
enabled: true
type: ClusterIP
port: 9090
targetPort: 9090
name: metrics
# Configuração do ServiceMonitor para o Prometheus Operator
# ref: https://github.com/prometheus-operator/prometheus-operator
serviceMonitor:
# Se true, um recurso ServiceMonitor será criado.
enabled: true
# O intervalo no qual as métricas devem ser coletadas (ex: 30s, 1m).
endpoints:
- port: metrics
path: /metrics
interval: 30s
relabelings: []
- port: sdk-metrics
path: /metrics
interval: 30s
relabelings: []
additionalLabels:
release: kube-prometheus-stack
ssh:
enabled: true
secretName: git-ssh-key-sientia-model-manager-worker
sshPath: /mnt/.ssh
knownHostsPath: /mnt/known_hosts
# Configuração para dashboards do Grafana
grafanaDashboard:
# Habilita a criação de ConfigMaps para dashboards
enabled: true
# Namespace onde o Grafana está instalado (ajuste conforme seu ambiente)
namespace: monitoring
# Labels para que o sidecar do Grafana encontre os dashboards
labels:
grafana_dashboard: "1"
# Lista de dashboards para importar
dashboards:
- name: sientia-dataops-model-manager
title: "Sientia DataOps Model Manager"
uid: "sientia-dataops-model-manager"
folder: "Sientia"
jsonFile: "dashboards/sientia-dataops-model-manager.json"
overwrite: true # Sobrescreve dashboard se já existir
version: "1.0.0" # Version inicial do dashboard
# Configuração para datasources do Grafana
grafanaDatasource:
# Habilita a criação de ConfigMap para datasources
enabled: false
# Namespace onde o Grafana está instalado
namespace: monitoring
# Labels para que o sidecar do Grafana encontre os datasources
labels:
grafana_datasource: "1"
# Lista de datasources para configurar
datasources: []
# Exemplo de datasource:
# - name: Prometheus
# type: prometheus
# url: http://prometheus-server.monitoring.svc.cluster.local
# isDefault: true
# jsonData:
# timeInterval: "5s"
# -----------------------------------------------------------------------------
# Helm usage examples
# -----------------------------------------------------------------------------
# kubectl create secret docker-registry docker-hub-secret --namespace sientia --docker-server=http://aignosi.azurecr.io --docker-username=aignosi --docker-password=<pwd>
#
# helm upgrade --install sientia-model-manager sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.6.1
#
# Global/runtimes layout note:
# - Shared configuration lives under `global` (env, probes, autoscaling, namespace).
# - Individual runtimes are defined under `runtimes`, each with its own `name` and `replicas`.
# - Runtimes inherit `global` settings unless overridden at the runtime level.
#
# kubectl create secret generic git-ssh-key-sientia-model-manager-worker \
# --namespace sientia \
# --from-file=ssh-privatekey=git_key \
# --type=kubernetes.io/ssh-auth
# kubectl create secret generic sientia-plugin-store-credentials \
# --namespace sientia \
# --from-literal=username=<username> \
# --from-literal=password=<password>