gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
221 Commits 2 Branches 0 Tags
9ea3249daf558a9f66098bfabf2bfd7e180f0687
Commit Graph

221 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Bruno Domingues
758ffb10b6 SIENTIAPDE-1241: Fix: Correctly handle feature normalization and denormalization, and add scaler parameters to metadata 2025-10-22 21:26:11 -03:00
Bruno Domingues
42a7e82c82 SIENTIAPDE-1241: Suppress sklearn FutureWarning regarding 'squared' deprecation. 2025-10-22 18:28:13 -03:00
Bruno Domingues
5789a13023 SIENTIAPDE-1241: refactor train_model workflow due to I/O errors. 2025-10-22 15:37:56 -03:00
Bruno Domingues
f2a1c88ff3 Merge pull request #8 from Aignosi/feature/SIENTIAPDE-1255 
SIENTIAPDE-1255: Refactor Model Manager for Training Pipeline, Integrate Sientia MLOps Library, and Add Unit Tests
 2025-10-21 09:42:17 -03:00
Bruno Domingues
ab7c93480b SIENTIAPDE-1255: Refactor: Move extra pip requirements to environment variable and use constants for preprocessor steps. 2025-10-21 09:20:23 -03:00
Bruno Domingues
4686b438db SIENTIAPDE-1255: Update quality gate workflow: Remove redundant repository and disk space cleanup, update cache key. 2025-10-20 22:35:20 -03:00
Bruno Domingues
3e52253a98 SIENTIAPDE-1255: Add tests to ensure get_required_columns removes columns from self_operations and cross_operations. 2025-10-20 22:27:58 -03:00
Bruno Domingues
614eb0cb7f SIENTIAPDE-1255: Add unit tests for utils.split_train_test function 2025-10-20 22:22:38 -03:00
Bruno Domingues
089f312575 SIENTIAPDE-1255: Add unit tests for the reports module 2025-10-20 22:20:26 -03:00
Bruno Domingues
0de2ece063 SIENTIAPDE-1255: Add unit tests for sientia models (DataPreprocessor and LinearRegressionModel) 2025-10-20 20:59:55 -03:00
Bruno Domingues
3c09e59f6a SIENTIAPDE-1255: Add unit tests for the ModelServing class. 2025-10-20 20:24:50 -03:00
Bruno Domingues
b1fe62f68b SIENTIAPDE-1255: Add unit tests for sientia metrics module (MAE, MSE, R2) 2025-10-20 20:13:08 -03:00
Bruno Domingues
b3e4bb8660 SIENTIAPDE-1255: Add comprehensive tests for __del__ methods in Activities and ExperimentTracking to ensure proper resource cleanup and exception handling. 2025-10-20 20:09:49 -03:00
Bruno Domingues
4a32d712ba SIENTIAPDE-1255: Implement object destructors for Activities and ExperimentTracking to prevent AttributeError during garbage collection, and mock workflow logger in tests to avoid RuntimeWarning. 2025-10-20 17:53:49 -03:00
Bruno Domingues
52be5c4e8e SIENTIAPDE-1255: Improve documentation for split_train_test function in utils.py 2025-10-20 17:41:05 -03:00
Bruno Domingues
0dbe179367 SIENTIAPDE-1255: Add thread-safety documentation and I/O notes to Reports class, improve save_all_sections documentation and error handling. 2025-10-20 17:38:44 -03:00
Bruno Domingues
2292ae57cc SIENTIAPDE-1255: Add thread-safety documentation and in-place modification warnings to LinearRegressionModel and DataPreprocessor classes. 2025-10-20 17:27:58 -03:00
Bruno Domingues
56f3db350c SIENTIAPDE-1255: Refactor ModelServing class to improve thread safety, resource management, and documentation. Adds context manager for experiment saving and clarifies thread-safety concerns. 2025-10-20 17:16:45 -03:00
Bruno Domingues
56a21a16da SIENTIAPDE-1255: Integrate sientia-mlops-library into model-manager, adding model serving, reporting, and updated model definitions. 2025-10-20 16:55:52 -03:00
Bruno Domingues
9e31ab679b SIENTIAPDE-1255: Implement MLFlow artifact management and model persistence 
This commit introduces a new model_repository.py to handle MLFlow artifact generation and model persistence. It also updates the README to reflect this change and modifies training_repository.py to separate training and MLFlow operations.
 2025-10-17 09:11:54 -03:00
Bruno Domingues
a66b996cc1 SIENTIAPDE-1255: Refactor MLFlow activities for training operations and update metrics 
This commit refactors the MLFlow activities to focus on model training rather than prediction operations. It removes prediction-related activities and metrics, and updates the MLFlow activity descriptions to reflect the change in focus. The README is also updated to reflect these changes.
 2025-10-17 00:51:29 -03:00
Bruno Domingues
85a8ba1e68 SIENTIAPDE-1255: Refactor data quality gates to training focused metrics and repositories. This commit removes the data quality gates and filters, focusing on training-specific metrics and data repositories. It also updates the README to reflect these changes, including new training metrics and a streamlined data services section. 2025-10-17 00:39:06 -03:00
Bruno Domingues
e191e7849f SIENTIAPDE-1255: Refactor Model Manager to focus on ML model training pipeline and simplify architecture. This includes removing prediction workflows, updating core functionality, and improving parameter validation and resource management. 
README.md | 411 deletions(-) 117 insertions(+)
1 file changed, 117 insertions(+), 411 deletions(-)
 2025-10-16 18:26:56 -03:00
Bruno Domingues
7abd951806 SIENTIAPDE-1255: Refactor worker to support only the train_model-queue and remove prediction workflows. 2025-10-16 18:15:52 -03:00
Bruno Domingues
8f1cc21bb1 SIENTIAPDE-1255: Add port forwarding setup script and documentation to README.md 2025-10-16 18:00:28 -03:00
Bruno Domingues
9ee12373b4 Merge pull request #7 from Aignosi/feature/SIENTIAPDE-1253 
SIENTIAPDE-1253: Improve Train Model Workflow with Enhanced Error Handling, Validation, and Cleanup
 2025-10-15 17:04:49 -03:00
Bruno Domingues
4ff796a56f SIENTIAPDE-1253: Update README.md to include documentation for the Train Model workflow and renumber the Minimal Retrain workflow. (+101, -1) 2025-10-15 16:35:13 -03:00
Bruno Domingues
98c216733b SIENTIAPDE-1253: Enhance error logging in train_model workflow with rich context for debugging. 2025-10-15 16:19:15 -03:00
Bruno Domingues
305a39b44c SIENTIAPDE-1253: Implement business rule validation for training parameters. Adds a validate_business_rules method to the TrainModelParams class to enforce constraints on training parameters, improving data integrity and preventing errors. Also updates tests to include target variable in variable columns and adds tests for business rule validations. 2025-10-15 16:16:09 -03:00
Bruno Domingues
4b28d5f1c8 SIENTIAPDE-1253: Implement granular retry policies for activities in train_model workflow. 2025-10-15 16:06:13 -03:00
Bruno Domingues
780184769d SIENTIAPDE-1253: Expose workflow activity timeouts as environment variables and document them. 2025-10-15 16:03:01 -03:00
Bruno Domingues
1ec40bfb2a SIENTIAPDE-1253: Implement activity for cleaning up temporary run directory and integrate into train model workflow. This change introduces a new activity for idempotent cleanup of the run directory after model training, replacing the direct directory removal in the workflow. This improves determinism and error handling. Also includes unit tests for the new activity. 2025-10-15 15:49:05 -03:00
Bruno Domingues
8ea98360c3 SIENTIAPDE-1253: Refactor training workflow and activities to raise exceptions on failure 
This commit refactors the training workflow and associated activities to raise exceptions on failure instead of returning success/failure dictionaries. This allows the Temporal workflow to handle errors more effectively and ensures that the workflow stops when a critical error occurs.

Key changes:

- The train_model workflow is introduced to orchestrate the entire training process, including parameter validation, data download, model training, and model saving.
- The validate_train_params activity is added to validate and convert training parameters.
- The train_model and save_model activities are updated to raise exceptions on failure.
- The ExperimentStatus enum is updated to include a new status for orchestrator validation errors.
- The tests are updated to reflect the new exception-based error handling.
- The activities now return the TrainModelResult directly instead of a dictionary.
 2025-10-15 15:19:29 -03:00
Bruno Domingues
61267ec49d SIENTIAPDE-1253: Enforce TrainModelParams object in Training activity and tests, removing dict conversion. 2025-10-14 10:03:34 -03:00
Bruno Domingues
014e1cd910 Merge pull request #6 from Aignosi/feature/SIENTIAPDE-1252 
SIENTIAPDE-1252: Implement Model Saving to MLflow and Artifact Logging
 2025-10-13 14:44:39 -03:00
Bruno Domingues
9d4ec19586 SIENTIAPDE-1252: Improve caching in quality gate, add alt text to logo, and change exception type in MLflow repository. 2025-10-13 10:54:38 -03:00
Bruno Domingues
b3c749872c SIENTIAPDE-1252: Implement save_model activity to save trained models to MLflow with comprehensive error handling and add corresponding unit tests. 2025-10-13 10:18:45 -03:00
Bruno Domingues
3fd5ab79fe SIENTIAPDE-1252: Implement artifact generation and MLflow logging for model training results 
This commit introduces artifact generation and MLflow logging capabilities to the model training process. It includes the following changes:

- Added methods to generate reports, save data files, and log model parameters, metrics, models, and artifacts to MLflow.
- Implemented error handling for various scenarios, such as missing files, invalid data, and MLflow connection errors.
- Created a new 'header.html' file for report styling and navigation.
- Modified the 'model_repository.py' file to include the new artifact generation and MLflow logging methods.
- Added comprehensive unit tests to ensure the functionality and robustness of the new features.
 2025-10-10 17:56:09 -03:00
Bruno Domingues
52b7b23261 SIENTIAPDE-1252: Update Python setup action and simplify dependency caching in quality gate workflow 2025-10-09 17:52:35 -03:00
Bruno Domingues
94e11df803 SIENTIAPDE-1252: Remove experiment_description from TrainModelParams and related tests. 2025-10-09 17:48:43 -03:00
Bruno Domingues
5b0cd92dfe Merge pull request #5 from Aignosi/feature/SIENTIAPDE-1251 
Implement ML Model Training Activity and Repository (SIENTIAPDE-1251)
 2025-10-09 17:09:56 -03:00
Bruno Domingues
bee6036205 SIENTIAPDE-1251: Implement ML model training activity and repository 
This commit introduces the 'Training' activity and 'TrainingRepository' for handling ML model training operations within the Model Manager system.

- Added model_manager/activities/training.py for the Training activity, which extends BaseActivity and integrates with Temporal workflows.
- Added model_manager/utils/repository/training_repository.py for the TrainingRepository, which encapsulates the core training logic.
- Updated model_manager/activities/activities.py to include the Training activity in the main activities orchestrator.
- Updated README.md to document the new 'Training' component.
- Added unit tests for the new activity and repository.
 2025-10-09 15:07:27 -03:00
Bruno Domingues
c970754e1b Merge pull request #4 from Aignosi/feature/SIENTIAPDE-1250 
SIENTIAPDE-1250: Implement Experiment Tracking Activity and Refactor Test Structure
 2025-10-07 14:51:14 -03:00
Bruno Domingues
97f5ee08f1 SIENTIAPDE-1250: Fix: Update pip cache key and replace pytz with datetime.UTC in experiment tracking. 2025-10-07 14:12:35 -03:00
Bruno Domingues
7e1eace77f SIENTIAPDE-1250: Implement experiment tracking activity with status updates, error handling, and model registration. This includes a unified update method, error message truncation, and integration into the main activities orchestrator. (236+, 2-) 2025-10-07 11:27:23 -03:00
Bruno Domingues
2628dc92d2 SIENTIAPDE-1250: Refactor: Remove 'laborious' directory from test structure and update README.md accordingly. 2025-10-07 10:02:38 -03:00
Bruno Domingues
851f34c80e Merge pull request #3 from Aignosi/feature/SIENTIAPDE-1249 
Implement data models for Model Manager
 2025-10-06 19:53:28 -03:00
Bruno Domingues
04c94efd98 SIENTIAPDE-1249: Refactor TrainModelParams to use dataclass and add from_dict method for validation, remove no-cache-dir from pip install in quality gate workflow, and rename X_train/X_test to x_train/x_test in TrainModelResult. 2025-10-06 19:11:02 -03:00
Bruno Domingues
d8583cdae7 SIENTIAPDE-1249: Implement data models for Model Manager and add unit tests. This commit introduces data transfer objects (DTOs) and model classes for experiment status, training parameters, and training results, along with corresponding unit tests to ensure their correct behavior. 2025-10-06 16:49:18 -03:00
Bruno Domingues
18ccfed78a Merge pull request #2 from Aignosi/feature/SIENTIAPDE-1248 
SIENTIAPDE-1248: Integrate MinIO, Add Worker Tests, and Refactor Documentation
 2025-10-06 11:27:19 -03:00