gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
215 Commits 2 Branches 0 Tags
c5cd382350dbd7749642f3820b7b2e93c5c4ae58
Commit Graph

104 Commits

Author SHA1 Message Date
vitor-aignosi
c5cd382350 feat: enhance configuration and scheduling for cleanup processes 
- Updated `.env.example` to include new environment variables for MinIO and PyPI configuration.
- Refactored `create_cleanup_schedule` to utilize runtime-specific task queues and improve schedule reconciliation logic.
- Enhanced `Activities` class to require a default bucket in MinIO configuration.
- Adjusted `requirements.txt` to specify version for `evidently`.
- Updated tests to reflect changes in schedule creation and configuration handling.
 2026-04-07 12:57:29 -03:00
vitor-aignosi
09ee92f100 feat: enhance configuration and error handling in project setup 
- Added new ignore rule for Ruff to allow temporary paths in tests.
- Introduced MyPy overrides for specific modules to ignore errors.
- Refactored `Cleanup` and `ExperimentTracking` classes to remove async keywords from methods, improving consistency in method signatures.
- Updated `Training` class methods to handle synchronous operations, enhancing performance and clarity.
- Adjusted `requirements.txt` to remove unnecessary Git dependency, streamlining project setup.
 2026-04-07 10:25:17 -03:00
vitor-aignosi
6b1df7c3a7 feat: enhance training and experiment tracking functionality 
- Updated `Activities` class to improve garbage collection handling.
- Enhanced error messaging in `ExperimentTracking` for better clarity on update failures.
- Refactored `Training` class to streamline exception handling and improve type hints.
- Introduced new methods in `TrainModelParams` for better handling of experiment run IDs and model metadata.
- Added functionality to extract model equations in `DataManagerRepository` for linear regression models.
 2026-04-06 15:05:57 -03:00
vitor-aignosi
d2bb588739 Merge branch 'main' into release/SIENTIAPDE-1645 2026-04-06 08:34:15 -03:00
Bruno Domingues
cce350bfb1 SIENTIAPDE-1717: Remove FILE_DELETE_ERROR from ExperimentStatus enum and related tests. 2026-03-30 14:33:19 -03:00
Bruno Domingues
f5eb8d1e5b SIENTIAPDE-1717: Remove list_bucket_objects method from StorageRepository and its associated test. 2026-03-30 14:18:14 -03:00
Bruno Domingues
7a0961f29d SIENTIAPDE-1717: Remove MinIO cleanup functionality and associated components. This change streamlines the cleanup workflow to focus solely on local temporary directories, removes the ModelTrainingError exception, and updates related configurations, documentation, and tests. 2026-03-30 14:14:10 -03:00
vitor-aignosi
342a02d6f7 feat: enhance training workflow with model metadata loading and refactor data handling 
- Introduced a new activity to load model metadata from the model store.
- Refactored training logic to utilize new model metadata and improved parameter handling.
- Updated the `TrainModelParams` class to include additional fields for model configuration.
- Replaced deprecated utility functions with a custom train-test split implementation.
- Removed unused utility functions and cleaned up the data manager repository.
- Adjusted experiment tracking to include model-specific metadata in notifications.
 2026-03-24 14:39:28 -03:00
vitor-aignosi
cf5111e520 feat: integrate PluginStore and MinIO repository into model manager activities 
- Added PluginStore integration for model management.
- Replaced StorageRepository with MinIORepository in Activities, Cleanup, and Training classes.
- Updated training logic to handle validation files and improved data management.
- Enhanced configuration for MinIO and PluginStore in connectors.
- Removed deprecated model repository and storage repository files.
- Updated environment variable handling for new configurations.
 2026-03-11 17:35:05 -03:00
Bruno Domingues
41c4490cad SIENTIAPDE-1579: Fix SientiaMlException propagation in ModelServing.search_runs_by_name to correctly raise the exception with its message, resolving a TypeError. Enhance training test script to support scenario-specific CSV files for different test cases. Update search_runs_by_name return type hint and apply minor code formatting. 2026-02-19 21:13:41 -03:00
Kou Kinoshita
14f3fbadbe SIENTIAPDE-1579: Switched from replace logic to validation + mapping 2026-02-18 18:57:41 -03:00
Kou Kinoshita
5dab1ee576 SIENTIAPDE-1579?: removed duplicated method 2026-02-18 18:18:39 -03:00
Kou Kinoshita
a7922ac3ce SIENTIAPDE-1579: Lint fix 2026-02-18 17:50:58 -03:00
Kou Kinoshita
665870320f SIENTIAPDE-1579: Updated tests and refactored apply filters method 2026-02-18 17:44:46 -03:00
Kou Kinoshita
43cbc31e10 SIENTIAPDE-1579: Lint fixes and formatting 2026-02-18 17:31:58 -03:00
Kou Kinoshita
c6d6f94e05 SIENTIAPDE-1579: Updated tests 2026-02-18 15:05:37 -03:00
Bruno Domingues
df6bf1daba SIENTIAPDE-1430: Refactor static threshold calculation logic and enhance test coverage 2025-12-19 16:19:10 -03:00
Bruno Domingues
06571011f2 SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal. 
This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null.
Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
 2025-12-19 15:44:18 -03:00
Bruno Domingues
38fb9a9c53 SIENTIAPDE-1430: Remove verbose options and direct print statements from model, preprocessor, and report utilities. 2025-12-18 21:09:12 -03:00
Bruno Domingues
13941fc3f7 SIENTIAPDE-1430: Add unit tests for _configure_datetime_index (handling various datetime column formats), _init_data_preprocessor (with removed intervals), and _extract_model_equation (supporting polynomial features) in TrainingRepository. 2025-12-18 18:58:12 -03:00
Bruno Domingues
71b654f24c SIENTIAPDE-1430: Add test case for ValueError when y_train_pred is None during artifact data initialization. 2025-12-18 18:48:42 -03:00
Bruno Domingues
573834fb99 SIENTIAPDE-1430: Add comprehensive tests for TrainModelParams.validate_business_rules to improve coverage. 2025-12-18 18:46:01 -03:00
Bruno Domingues
70067fd992 SIENTIAPDE-1430: Improve test coverage for LinearRegressionModel and DataPreprocessor. Added extensive new tests covering various methods, edge cases, and scenarios to enhance overall code coverage. 2025-12-18 18:41:03 -03:00
Bruno Domingues
0d50a76f4c SIENTIAPDE-1430: Add unit tests for Silverman radius and RCE drift metrics. 2025-12-18 18:19:35 -03:00
Bruno Domingues
06fd08dc70 SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations. 2025-12-18 17:05:10 -03:00
Bruno Domingues
6e8f87b2a3 SIENTIAPDE-1430: Implement advanced model training capabilities and enhanced data preprocessing. This includes support for Polynomial Regression with configurable degree and interaction terms, flexible per-variable lag configurations, and new data filtering options by date range and removed intervals. Comprehensive business validations are now enforced for all parameters, and MLflow logging has been extended to capture these detailed configurations. Additionally, Reduced Coulomb Energy (RCE) metrics are added for drift detection, with a new changelog documenting all pipeline parameter updates. 2025-12-17 21:39:43 -03:00
Bruno Domingues
5dcb62ef4b SIENTIAPDE-1350: Integrate cleanup workflow and update default task queue names. 2025-11-26 22:11:20 -03:00
Bruno Domingues
11aaa0780f SIENTIAPDE-1350: Improve Docker image build context and fix queue names 
This commit refactors the .dockerignore file to use a whitelist approach, significantly reducing the Docker image size by excluding unnecessary files. It also fixes the queue names in values.yaml and adds TEMPORAL_USE_TLS to the test environment.
 2025-11-26 17:52:44 -03:00
Bruno Domingues
c281969381 SIENTIAPDE-1350: Add test to ensure worker starts despite cleanup schedule creation failure. 2025-11-26 11:30:22 -03:00
Bruno Domingues
bc917e97d8 SIENTIAPDE-1350: Add cleanup schedule tests and configure worker with task queues from env vars 2025-11-25 19:21:55 -03:00
Bruno Domingues
b028dc1b7d SIENTIAPDE-1350: Implement scheduled cleanup workflow using Temporal schedules. Adds schedule creation to worker startup and configures environment variables. 2025-11-25 18:12:53 -03:00
Bruno Domingues
823df2d65d SIENTIAPDE-1350: Refactor: Replace AsyncMock with MagicMock in cleanup tests for synchronous calls 2025-11-25 16:54:45 -03:00
Bruno Domingues
33480fa108 SIENTIAPDE-1350: Add unit tests for the CleanupFiles workflow, covering both input bucket and default bucket scenarios. 2025-11-25 16:44:51 -03:00
Bruno Domingues
d0b5b74f88 SIENTIAPDE-1350: Add tests for close method and paginated list_bucket_objects 2025-11-25 16:04:59 -03:00
Bruno Domingues
d2c4803a75 SIENTIAPDE-1350: Add unit tests for the Cleanup activity, ensuring 100% code coverage, and update todo-list. 2025-11-25 16:01:10 -03:00
Bruno Domingues
f3c88885ea SIENTIAPDE-1350: Configure cleanup test script and improve test isolation. This commit configures the cleanup test script to load environment variables from a .env file and use them for Temporal connection. It also adds a fixture to clean up temporary directories created by tests, ensuring better test isolation and preventing potential conflicts. Additionally, it adds the 'uri' property to the MongoDB config. 2025-11-25 00:44:55 -03:00
Bruno Domingues
2ee66552d2 SIENTIAPDE-1350: Configure separate task queues for train and cleanup workers, update sientia-dataops-library to 1.6.1 and refactor prometheus server startup to use logger. 2025-11-24 11:21:43 -03:00
Bruno Domingues
53f49d7a97 SIENTIAPDE-1350: Implement file cleanup workflow and activities, including MinIO and local directory cleanup, configuration, and metrics. 2025-11-19 18:38:00 -03:00
Bruno Domingues
7ff54a1112 SIENTIAPDE-1309: Improve Dockerignore, Gitignore, Dockerfile, README, model repository, tests, todo list and values.yaml files. 2025-11-07 15:08:50 -03:00
Bruno Domingues
c7f44a2423 SIENTIAPDE-1309: Update README with Helm instructions and refactor experiment status messages. Also, update values.yaml with new image and configurations. 2025-11-06 15:34:03 -03:00
Bruno Domingues
f32d647d1c SIENTIAPDE-1307: Implement metrics for training activities and workflow executions, tracking success/failure status. 2025-11-03 14:41:37 -03:00
Bruno Domingues
d8e8d3c481 SIENTIAPDE-1307: Fix PytestUnraisableExceptionWarning in activity and experiment tracking tests by controlling exception raising in __del__ mocks. 2025-10-31 17:21:27 -03:00
Bruno Domingues
a258036397 SIENTIAPDE-1307: Refactor: Remove redundant patching of SientiaMonitoring and Postgres init in tests, and add shutdown mock to metrics controller. 2025-10-31 17:15:59 -03:00
Bruno Domingues
db581f60f9 SIENTIAPDE-1307: Integrate metrics controller and update sientia-dataops-library to 1.5.1. This change adds metrics collection capabilities to activities and updates the dataops library dependency. (18 files changed, 143 insertions(+), 16 deletions(-)) 2025-10-31 17:06:26 -03:00
Bruno Domingues
5b14f1016b SIENTIAPDE-1241: Suppress PytestUnraisableExceptionWarning and update test data in training repository tests. 2025-10-30 15:22:59 -03:00
Bruno Domingues
f5c1ac3615 SIENTIAPDE-1241: Remove model equation implementation documentation and add test coverage for _generate_artifacts in ModelRepository. 2025-10-30 14:56:10 -03:00
Kou-Kinoshita
a5c0cb6e47 SIENTIAPDE-1241: Fixed formatting and linted code 2025-10-30 10:38:29 -03:00
Kou-Kinoshita
17a551d6f9 SIENTIAPDE-1241: Added remaining training repository tests 2025-10-30 10:29:51 -03:00
Kou-Kinoshita
7e9d6683e4 SIENTIAPDE-1241: Added worker test 2025-10-30 10:21:39 -03:00
Kou-Kinoshita
c63389620a SIENTIAPDE-1241: Train_model tests 2025-10-30 10:11:07 -03:00