gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SIENTIAPDE-1352: Update README with improved instructions, project structure details, and environment variable descriptions.

Browse Source
This commit is contained in:
Bruno Domingues
2025-12-04 19:31:35 -03:00
parent 01b8c06d75
commit 32aa66e5d8
1 changed files with 92 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

158
README.md
View File

@@ -669,27 +669,27 @@ chmod +x run_local.sh
```
The script will:
- Activate the virtual environment
- Load environment variables from `.env`
- Start the model-manager worker application
### Running Tests and Coverage
Use the provided script to run tests with coverage:
**Note**: Activate your virtual environment before running the script:
```bash
# Make script executable (first time only)
chmod +x run_coverage.sh
# Run tests with coverage
./run_coverage.sh
source ./venv/bin/activate # or: conda activate ./venv
./run_local.sh
```
The script will:
- Activate the virtual environment
- Run pytest with coverage reporting
- Generate HTML coverage report
- Open the coverage report in your browser
### Running Tests and Coverage
Run tests with coverage using pytest directly:
```bash
# Run all tests with coverage
pytest tests/ --cov=model_manager --cov-report=term-missing --cov-report=html
# Open the coverage report in your browser
open htmlcov/index.html # macOS
xdg-open htmlcov/index.html # Linux
```
### Manual Test Execution
@@ -852,8 +852,10 @@ Runs automatically when a PR is merged to `main`:
tests/
├── activities/ # Activity implementation tests
├── workflows/ # Workflow orchestration tests
├── utils/ # Utility function tests
└── worker/ # Worker tests
├── utils/ # Utility function tests
├── worker/ # Worker tests
├── schedules/ # Schedule configuration tests
└── sientia/ # Sientia module tests
```
### Test Execution
@@ -912,8 +914,7 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
| `POSTGRES_DBNAME` | PostgreSQL database | `sientia` | Yes |
| `POSTGRES_MIN_CONNECTIONS` | Minimum PostgreSQL connections | `5` | No |
| `POSTGRES_MAX_CONNECTIONS` | Maximum PostgreSQL connections | `20` | No |
| `MLFLOW_HOST` | MLFlow server hostname | `http://localhost` | Yes |
| `MLFLOW_PORT` | MLFlow server port | `5080` | Yes |
| `MLFLOW_URL` | MLFlow server URL (full URL with protocol and port) | `http://localhost:5080` | Yes |
| `MLFLOW_USERNAME` | MLFlow username | `aignosi` | Yes |
| `MLFLOW_PASSWORD` | MLFlow password | `aignosi` | Yes |
| `MINIO_ENDPOINT_URL` | MinIO server endpoint | `http://minio.minio.svc.cluster.local:9000` | Yes |
@@ -928,7 +929,7 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
| `MONGODB_URL` | MongoDB connection URI | `localhost:27018` | Yes |
| `MONGODB_USERNAME` | MongoDB username | `root` | Yes |
| `MONGODB_PASSWORD` | MongoDB password | `wKZDbMNU1c` | Yes |
| `MONGODB_DATABASE_NAME` | MongoDB database name | `sientia` | Yes |
| `MONGODB_DATABASE` | MongoDB database name | `sientia` | Yes |
| `MONGODB_TTL_INDEX_HOURS` | MongoDB TTL index hours | `1` | No |
| `CLEANUP_SCHEDULE_ID` | Cleanup schedule identifier | `cleanup-files-daily` | No |
| `CLEANUP_CRON` | Cleanup cron expression | `0 0 * * *` | No |
@@ -943,6 +944,7 @@ The Model Manager system exposes comprehensive Prometheus metrics for operationa
| `HTTP_METRICS_PORT` | Prometheus metrics port | `9090` | No |
| `HTTP_SDK_METRICS_PORT` | Temporal SDK metrics port | `9091` | No |
| `POD_ID` | Kubernetes pod identifier | `None` | No |
| `EXTRA_PIP_REQUIREMENTS` | Extra pip requirements for MLFlow model serving | `None` | No |
#### Workflow Activity Timeouts
@@ -1036,51 +1038,73 @@ Version is calculated automatically in the CI/CD pipeline and passed to SonarQub
### Project Structure
```
model_manager/
├── activities/ # Temporal activity implementations
│ ├── __init__.py
│ ├── activities.py # Main activities orchestrator (combines all activities)
│ ├── experiment_tracking.py # Experiment status tracking and database operations
│ ├── training.py # ML model training operations (includes MLFlow & MinIO)
│ └── cleanup.py # File and directory cleanup operations
├── workflows/ # Temporal workflow definitions
│ ├── __init__.py
│ ├── train_model.py # Complete ML model training workflow
│ └── cleanup_files.py # Automated file cleanup workflow
├── schedules/ # Temporal schedule configurations
│ ├── __init__.py
│ └── cleanup_schedule.py # Cleanup schedule creation and management
├── worker/ # Worker implementation
│ ├── __init__.py
│ └── worker.py # Main worker orchestrator (Temporal client setup)
├── utils/ # Utility functions and helpers
│ ├── __init__.py
│ ├── connectors_config.py # Environment-based configuration builders
│ ├── exceptions.py # Custom exception definitions
│ ├── logger_helper.py # Logger initialization utilities
│ ├── models/ # Data models and schemas
sientia-dataops-model-manager/
├── model_manager/ # Main application package
│ ├── activities/ # Temporal activity implementations
│ │ ├── __init__.py
│ │ ├── train_model_params.py # Training parameters model
│ │ ├── train_model_result.py # Training result model
│ │ └── experiment_status.py # Experiment status enum
│ └── repository/ # Data access layer
│ ├── __init__.py
│ ├── training_repository.py # Training business logic
│ ├── model_repository.py # MLFlow artifact management
│ └── storage_repository.py # MinIO storage operations
├── sientia/ # Sientia-specific implementations
│ ├── __init__.py
│ ├── exceptions.py # Custom exceptions
│ ├── metrics.py # Business metrics
│ ├── models.py # ML model implementations
│ ├── model_serving.py # Model serving utilities
│ ├── reports.py # Report generation
│ └── utils.py # Utility functions
├── reports/ # Report templates and temporary files
│ ├── header.html # HTML report header template
│ └── temp/ # Temporary report files (cleaned up automatically)
├── metrics.py # Prometheus metrics definitions
└── __init__.py
│ │ ├── activities.py # Main activities orchestrator (combines all activities)
│ │ ├── experiment_tracking.py # Experiment status tracking and database operations
│ │ ├── training.py # ML model training operations (includes MLFlow & MinIO)
│ │ └── cleanup.py # File and directory cleanup operations
│ ├── workflows/ # Temporal workflow definitions
│ │ ├── __init__.py
│ │ ├── train_model.py # Complete ML model training workflow
│ │ └── cleanup_files.py # Automated file cleanup workflow
│ ├── schedules/ # Temporal schedule configurations
│ │ ├── __init__.py
│ │ └── cleanup_schedule.py # Cleanup schedule creation and management
│ ├── worker/ # Worker implementation
│ │ ├── __init__.py
│ │ └── worker.py # Main worker orchestrator (Temporal client setup)
│ ├── utils/ # Utility functions and helpers
│ │ ├── __init__.py
│ │ ├── connectors_config.py # Environment-based configuration builders
│ │ ├── exceptions.py # Custom exception definitions
│ │ ├── logger_helper.py # Logger initialization utilities
│ │ ├── models/ # Data models and schemas
│ │ │ ├── __init__.py
│ │ │ ├── train_model_params.py # Training parameters model
│ │ │ ├── train_model_result.py # Training result model
│ │ │ └── experiment_status.py # Experiment status enum
│ │ └── repository/ # Data access layer
│ │ ├── training_repository.py # Training business logic
│ │ ├── model_repository.py # MLFlow artifact management
│ │ └── storage_repository.py # MinIO storage operations
│ ├── sientia/ # Sientia-specific implementations
│ │ ├── __init__.py
│ │ ├── exceptions.py # Custom exceptions
│ │ ├── metrics.py # Business metrics
│ │ ├── models.py # ML model implementations
│ │ ├── model_serving.py # Model serving utilities
│ │ ├── reports.py # Report generation
│ │ └── utils.py # Utility functions
│ ├── reports/ # Report templates and temporary files
│ │ ├── header.html # HTML report header template
│ │ └── temp/ # Temporary report files (cleaned up automatically)
│ ├── metrics.py # Prometheus metrics definitions
│ └── __init__.py
├── scripts/ # Test and utility scripts
│ ├── run_cleanup_test.py # Manual cleanup workflow test
│ └── run_training_test.py # Manual training workflow test
├── tests/ # Test suite
│ ├── activities/ # Activity tests
│ ├── workflows/ # Workflow tests
│ ├── utils/ # Utility tests
│ ├── worker/ # Worker tests
│ ├── schedules/ # Schedule tests
│ └── sientia/ # Sientia module tests
├── docs/ # Documentation and test data
├── .github/workflows/ # CI/CD workflows
│ ├── quality-gate.yml # PR quality checks
│ └── deploy.yml # Deployment workflow
├── Dockerfile # Container image definition
├── values.yaml # Helm chart values
├── pyproject.toml # Project configuration
├── requirements.txt # Production dependencies
├── requirements-dev.txt # Development dependencies
├── validate.sh # Code quality validation script
├── run_local.sh # Local execution script
└── README.md # This file
```
### Adding New Features
@@ -1104,8 +1128,10 @@ model_manager/
tests/
├── activities/ # Activity tests
├── workflows/ # Workflow tests
├── utils/ # Utility tests
└── worker/ # Worker tests
├── utils/ # Utility tests
├── worker/ # Worker tests
├── schedules/ # Schedule tests
└── sientia/ # Sientia module tests
```
## Troubleshooting