SIENTIAPDE-1243: Rename project from 'laborious' to 'model-manager' across codebase and configuration. This includes updating project names in environment variables, Makefiles, README, metrics, and Helm chart values.
This commit is contained in:
@@ -17,10 +17,10 @@ OPC_URL="opc.tcp://sientia-opc-simulator-opc.sientia.svc.cluster.local:4840"
|
||||
LOG_LEVEL="DEBUG"
|
||||
HTTP_METRICS_PORT="9090"
|
||||
HTTP_SDK_METRICS_PORT="9091"
|
||||
PROJECT_NAME="sientia-laborious"
|
||||
PROJECT_NAME="sientia-model-manager"
|
||||
|
||||
TEMPORAL_HOST="temporal-frontend.temporal.svc.cluster.local:7233"
|
||||
TEMPORAL_NAMESPACE="laborious"
|
||||
TEMPORAL_NAMESPACE="model-manager"
|
||||
|
||||
MONGODB_USERNAME="mongo_user"
|
||||
MONGODB_PASSWORD="mongo_db_password"
|
||||
|
||||
2
Makefile
2
Makefile
@@ -1,5 +1,5 @@
|
||||
VERSION = 1.0.8
|
||||
name = sientia-laborious
|
||||
name = sientia-model-manager
|
||||
# ENVIRONMENT = production
|
||||
|
||||
docker-hub:
|
||||
|
||||
26
README.md
26
README.md
@@ -21,7 +21,7 @@ A comprehensive AI model management platform for the complete machine learning l
|
||||
|
||||
## Architecture
|
||||
|
||||
The Laborious system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
|
||||
The Model Manager system uses a Temporal-based workflow architecture with clear separation of concerns and robust error handling. The architecture is designed for high availability, scalability, and operational excellence in production ML environments.
|
||||
|
||||
|
||||
### Architecture Principles
|
||||
@@ -428,7 +428,7 @@ flowchart LR
|
||||
|
||||
## 📦 How to Run
|
||||
|
||||
### Running the Laborious Application
|
||||
### Running the Model Manager Application
|
||||
|
||||
Use the provided script to run the application locally:
|
||||
|
||||
@@ -443,7 +443,7 @@ chmod +x run_local.sh
|
||||
The script will:
|
||||
- Activate the virtual environment
|
||||
- Load environment variables from `.env`
|
||||
- Start the laborious worker application
|
||||
- Start the model-manager worker application
|
||||
|
||||
### Running Tests and Coverage
|
||||
|
||||
@@ -495,7 +495,7 @@ if [ -f .env ]; then
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
fi
|
||||
|
||||
# Start the laborious worker
|
||||
# Start the model-manager worker
|
||||
python -m model_manager.worker.worker
|
||||
```
|
||||
|
||||
@@ -525,25 +525,25 @@ pytest tests/workflow/test_predictions_batch.py
|
||||
|
||||
## 📊 Monitoring and Metrics
|
||||
|
||||
The Laborious system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:
|
||||
The Model Manager system exposes comprehensive Prometheus metrics for operational visibility and performance monitoring:
|
||||
|
||||
### Application Health Metrics
|
||||
- `app_up`: Application health status (1=healthy, 0=unhealthy)
|
||||
- Labels: `pod_id`
|
||||
|
||||
### Prediction Operation Metrics
|
||||
- `laborious_predictions_written_count`: Counter for successful prediction exports
|
||||
- `model_manager_predictions_written_count`: Counter for successful prediction exports
|
||||
- Labels: `pod_id`, `model_name`, `pipeline_name`
|
||||
- `laborious_prediction_confidence_monitor`: Gauge for current prediction confidence levels
|
||||
- `model_manager_prediction_confidence_monitor`: Gauge for current prediction confidence levels
|
||||
- Labels: `pod_id`, `model_name`, `pipeline_name`
|
||||
- `laborious_prediction_response_time_monitor`: Histogram for prediction response times
|
||||
- `model_manager_prediction_response_time_monitor`: Histogram for prediction response times
|
||||
- Labels: `pod_id`, `model_name`, `pipeline_name`
|
||||
- Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]
|
||||
|
||||
### OPC Export Metrics
|
||||
- `laborious_prediction_opc_writing_count`: Counter for OPC server write operations
|
||||
- `model_manager_prediction_opc_writing_count`: Counter for OPC server write operations
|
||||
- Labels: `pod_id`, `model_name`, `pipeline_name`, `opc_server_id`
|
||||
- `laborious_prediction_opc_writing_response_time_monitor`: Histogram for OPC write response times
|
||||
- `model_manager_prediction_opc_writing_response_time_monitor`: Histogram for OPC write response times
|
||||
- Labels: `pod_id`, `model_name`, `pipeline_name`, `opc_server_id`
|
||||
- Buckets: [0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]
|
||||
|
||||
@@ -559,7 +559,7 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
|
||||
| Variable | Description | Default | Required |
|
||||
|----------|-------------|---------|----------|
|
||||
| `TEMPORAL_HOST` | Temporal server address | `localhost:7233` | Yes |
|
||||
| `TEMPORAL_NAMESPACE` | Temporal namespace | `laborious` | No |
|
||||
| `TEMPORAL_NAMESPACE` | Temporal namespace | `model-manager` | No |
|
||||
| `POSTGRES_HOST` | PostgreSQL hostname | `localhost` | Yes |
|
||||
| `POSTGRES_PORT` | PostgreSQL port | `5432` | Yes |
|
||||
| `POSTGRES_USER` | PostgreSQL username | `sientia` | Yes |
|
||||
@@ -585,7 +585,7 @@ The Laborious system exposes comprehensive Prometheus metrics for operational vi
|
||||
| `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes |
|
||||
| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index hours | `1` | No |
|
||||
| `LOG_LEVEL` | Application log level | `INFO` | No |
|
||||
| `PROJECT_NAME` | Project name for metrics | `laborious` | No |
|
||||
| `PROJECT_NAME` | Project name for metrics | `model-manager` | No |
|
||||
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
|
||||
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
|
||||
| `POD_ID` | Kubernetes pod identifier | `None` | No |
|
||||
@@ -838,4 +838,4 @@ For support and questions:
|
||||
|
||||
---
|
||||
|
||||
**Note**: The Laborious system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.
|
||||
**Note**: The Model Manager system is designed for production use in industrial ML environments. Ensure proper security configuration and network isolation for production deployments.
|
||||
|
||||
@@ -12,7 +12,7 @@ with workflow.unsafe.imports_passed_through():
|
||||
|
||||
class Activities(Postgres, MLFlow, Gates, OPC):
|
||||
"""
|
||||
Main activities orchestrator for the Laborious system.
|
||||
Main activities orchestrator for the Model Manager system.
|
||||
|
||||
This class combines functionality from multiple activity classes to provide
|
||||
a unified interface for all workflow operations. It manages database connections,
|
||||
|
||||
@@ -53,7 +53,7 @@ mlflow_content_filter_functions = {
|
||||
|
||||
class Gates(BaseActivity):
|
||||
"""
|
||||
Data quality gates and filtering activities for the Laborious system.
|
||||
Data quality gates and filtering activities for the Model Manager system.
|
||||
|
||||
This class implements comprehensive data quality validation and filtering
|
||||
mechanisms that can be applied at different stages of the prediction pipeline.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
Laborious Metrics Module
|
||||
Model Manager Metrics Module
|
||||
|
||||
This module defines all Prometheus metrics used by the Sientia DataOps Laborious system
|
||||
This module defines all Prometheus metrics used by the Sientia DataOps Model Manager system
|
||||
for monitoring and observability. The metrics provide insights into system performance,
|
||||
prediction quality, and operational health.
|
||||
|
||||
@@ -37,21 +37,21 @@ CORE_LABELS = ["pod_id", "model_name", "pipeline_name"]
|
||||
|
||||
# Prediction operation metrics
|
||||
PREDICTIONS_WRITTEN_COUNT = Counter(
|
||||
"laborious_predictions_written_count",
|
||||
"model_manager_predictions_written_count",
|
||||
"Number of predictions written to the database table predictions",
|
||||
CORE_LABELS,
|
||||
)
|
||||
|
||||
# Prediction quality metrics
|
||||
PREDICTION_CONFIDENCE_MONITOR = Gauge(
|
||||
"laborious_prediction_confidence_monitor",
|
||||
"model_manager_prediction_confidence_monitor",
|
||||
"Current confidence of each prediction",
|
||||
CORE_LABELS,
|
||||
)
|
||||
|
||||
# Performance monitoring metrics
|
||||
PREDICTION_RESPONSE_TIME_MONITOR = Histogram(
|
||||
"laborious_prediction_response_time_monitor",
|
||||
"model_manager_prediction_response_time_monitor",
|
||||
"Current response time of each prediction",
|
||||
CORE_LABELS,
|
||||
buckets=[0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]
|
||||
@@ -59,13 +59,13 @@ PREDICTION_RESPONSE_TIME_MONITOR = Histogram(
|
||||
|
||||
# OPC export metrics
|
||||
PREDICTION_OPC_WRITING_COUNT = Counter(
|
||||
"laborious_prediction_opc_writing_count",
|
||||
"model_manager_prediction_opc_writing_count",
|
||||
"Number of predictions written to the OPC server",
|
||||
[*CORE_LABELS, "opc_server_id"],
|
||||
)
|
||||
|
||||
PREDICTION_OPC_WRITING_RESPONSE_TIME_MONITOR = Histogram(
|
||||
"laborious_prediction_opc_writing_response_time_monitor",
|
||||
"model_manager_prediction_opc_writing_response_time_monitor",
|
||||
"Current response time of each prediction written to the OPC server",
|
||||
[*CORE_LABELS, "opc_server_id"],
|
||||
buckets=[0.01, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0]
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""
|
||||
Laborious Worker Module
|
||||
Model Manager Worker Module
|
||||
|
||||
This module provides the main worker implementation for the Sientia DataOps Laborious system.
|
||||
This module provides the main worker implementation for the Sientia DataOps Model Manager system.
|
||||
It orchestrates Temporal workers, manages task queues, and handles the lifecycle of
|
||||
prediction and retraining workflows.
|
||||
|
||||
@@ -18,11 +18,11 @@ Key Features:
|
||||
|
||||
Environment Variables:
|
||||
- TEMPORAL_HOST: Temporal server address (default: localhost:7233)
|
||||
- TEMPORAL_NAMESPACE: Temporal namespace (default: laborious)
|
||||
- TEMPORAL_NAMESPACE: Temporal namespace (default: model_manager)
|
||||
- POD_ID: Kubernetes pod identifier for metrics
|
||||
- HTTP_METRICS_PORT: Prometheus metrics server port (default: 9090)
|
||||
- HTTP_SDK_METRICS_PORT: Temporal SDK metrics port (default: 9091)
|
||||
- PROJECT_NAME: Project name for notifications (default: laborious)
|
||||
- PROJECT_NAME: Project name for notifications (default: model_manager)
|
||||
"""
|
||||
|
||||
from temporalio import workflow, client
|
||||
@@ -56,7 +56,7 @@ SDK_METRICS_PORT = int(os.getenv('HTTP_SDK_METRICS_PORT', "9091"))
|
||||
|
||||
async def main():
|
||||
"""
|
||||
Main entry point for the Laborious worker application.
|
||||
Main entry point for the Model Manager worker application.
|
||||
|
||||
This function initializes and starts all components of the worker:
|
||||
1. Sets up logging and metadata
|
||||
@@ -97,7 +97,7 @@ async def main():
|
||||
connection_string=mongo_config['connection_string'],
|
||||
database=mongo_config['database_name'],
|
||||
logger=logger,
|
||||
project_name=os.getenv('PROJECT_NAME', 'laborious')
|
||||
project_name=os.getenv('PROJECT_NAME', 'model-manager')
|
||||
)
|
||||
|
||||
logger.custom_info('Starting Activities...', metadata)
|
||||
@@ -127,7 +127,7 @@ async def main():
|
||||
|
||||
temporal_client = await client.Client.connect(
|
||||
target_host=host,
|
||||
namespace=os.getenv('TEMPORAL_NAMESPACE', 'laborious'),
|
||||
namespace=os.getenv('TEMPORAL_NAMESPACE', 'model-manager'),
|
||||
runtime=new_runtime
|
||||
)
|
||||
|
||||
@@ -214,7 +214,7 @@ def start_prometheus_server():
|
||||
and sets the application health metric to indicate the service is running.
|
||||
|
||||
The server exposes metrics that can be scraped by Prometheus for monitoring
|
||||
the health and performance of the Laborious worker.
|
||||
the health and performance of the Model Manager worker.
|
||||
|
||||
Environment Variables:
|
||||
HTTP_METRICS_PORT: Port for the metrics server (default: 9090)
|
||||
|
||||
@@ -10,7 +10,7 @@ with workflow.unsafe.imports_passed_through():
|
||||
@workflow.defn(name="minimal_retrain")
|
||||
class MinimalRetrain():
|
||||
"""
|
||||
Automated model retraining workflow for the Laborious system.
|
||||
Automated model retraining workflow for the Model Manager system.
|
||||
|
||||
This workflow implements a complete model retraining pipeline that loads
|
||||
training data, executes model retraining, updates production models,
|
||||
|
||||
@@ -10,7 +10,7 @@ with workflow.unsafe.imports_passed_through():
|
||||
@workflow.defn(name="predictions_batch")
|
||||
class PredictionsBatch():
|
||||
"""
|
||||
Main batch prediction workflow for the Laborious system.
|
||||
Main batch prediction workflow for the Model Manager system.
|
||||
|
||||
This workflow orchestrates the complete batch prediction process, handling
|
||||
data loading, configuration management, and workflow delegation. It serves
|
||||
|
||||
@@ -10,7 +10,7 @@ with workflow.unsafe.imports_passed_through():
|
||||
@workflow.defn(name="prediction_process")
|
||||
class PredictionProcess():
|
||||
"""
|
||||
Core prediction processing workflow for the Laborious system.
|
||||
Core prediction processing workflow for the Model Manager system.
|
||||
|
||||
This workflow implements the complete ML model inference pipeline, handling
|
||||
data quality validation, MLFlow model interactions, and prediction processing.
|
||||
|
||||
18
values.yaml
18
values.yaml
@@ -17,8 +17,8 @@ image:
|
||||
imagePullSecrets:
|
||||
- name: docker-hub-secret
|
||||
# This is to override the chart name.
|
||||
nameOverride: "sientia-laborious-worker"
|
||||
fullnameOverride: "sientia-laborious-worker"
|
||||
nameOverride: "sientia-model-manager-worker"
|
||||
fullnameOverride: "sientia-model-manager-worker"
|
||||
namespace: sientia
|
||||
|
||||
# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
|
||||
@@ -31,7 +31,7 @@ serviceAccount:
|
||||
annotations: {}
|
||||
# The name of the service account to use.
|
||||
# If not set and create is true, a name is generated using the fullname template
|
||||
name: "sientia-laborious-worker"
|
||||
name: "sientia-model-manager-worker"
|
||||
|
||||
# This is for setting Kubernetes Annotations to a Pod.
|
||||
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
|
||||
@@ -149,7 +149,7 @@ serviceMonitor:
|
||||
env:
|
||||
# Entrypoint variables
|
||||
- name: GITHUB_REPO_URL
|
||||
value: "git@github.com:Aignosi/sientia-dataops-laborious_temporal.git"
|
||||
value: "git@github.com:Aignosi/sientia-dataops-model-manager.git"
|
||||
- name: GITHUB_BRANCH
|
||||
value: SIENTIAPDE-1222-ajustar-a-library-para-fazer-o-download-do-courier
|
||||
- name: PYTHON_APP
|
||||
@@ -195,12 +195,12 @@ env:
|
||||
- name: HTTP_SDK_METRICS_PORT
|
||||
value: "9091"
|
||||
- name: PROJECT_NAME
|
||||
value: "sientia-laborious"
|
||||
value: "sientia-model-manager"
|
||||
|
||||
- name: TEMPORAL_HOST
|
||||
value: "temporal-frontend.temporal.svc.cluster.local:7233"
|
||||
- name: TEMPORAL_NAMESPACE
|
||||
value: "laborious"
|
||||
value: "model-manager"
|
||||
|
||||
- name: MONGODB_USERNAME
|
||||
value: "root"
|
||||
@@ -215,15 +215,15 @@ env:
|
||||
|
||||
ssh:
|
||||
enabled: true
|
||||
secretName: git-ssh-key-sientia-laborious-worker
|
||||
secretName: git-ssh-key-sientia-model-manager-worker
|
||||
sshPath: /mnt/.ssh
|
||||
knownHostsPath: /mnt/known_hosts
|
||||
|
||||
# kubectl create secret docker-registry docker-hub-secret --namespace sientia --docker-server=http://aignosi.azurecr.io --docker-username=aignosi --docker-password=5I5zpQ6sRaHqX1hD3dr+2mo647yO3FRc359/wu6gsP+ACRDRz5mp
|
||||
|
||||
# helm upgrade --install sientia-laborious-worker sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.5.0
|
||||
# helm upgrade --install sientia-model-manager-worker sientia/sientia-module -n sientia --create-namespace -f ./values.yaml --version 0.5.0
|
||||
|
||||
# kubectl create secret generic git-ssh-key-sientia-laborious-worker \
|
||||
# kubectl create secret generic git-ssh-key-sientia-model-manager-worker \
|
||||
# --namespace sientia \
|
||||
# --from-file=ssh-privatekey=git_key \
|
||||
# --type=kubernetes.io/ssh-auth
|
||||
|
||||
Reference in New Issue
Block a user