Commit Graph

213 Commits

Author SHA1 Message Date
vitor-aignosi
6b1df7c3a7 feat: enhance training and experiment tracking functionality
- Updated `Activities` class to improve garbage collection handling.
- Enhanced error messaging in `ExperimentTracking` for better clarity on update failures.
- Refactored `Training` class to streamline exception handling and improve type hints.
- Introduced new methods in `TrainModelParams` for better handling of experiment run IDs and model metadata.
- Added functionality to extract model equations in `DataManagerRepository` for linear regression models.
2026-04-06 15:05:57 -03:00
vitor-aignosi
1352d1ac8f feat: enhance training model functionality and reporting
- Added `evidently` to requirements for improved model evaluation.
- Introduced `TrainModelResult` class with a `to_dict` method for better result handling.
- Updated `train_model` method to return a comprehensive training result, including run details.
- Enhanced `cleanup_run_directory` method in `DataManagerRepository` for improved resource management.
- Adjusted type hints in `TrainModel` for clarity and consistency.
2026-04-06 11:40:08 -03:00
vitor-aignosi
a8b926649a feat: enhance training reporting and directory management
- Added functionality to store run name and ID in the training results.
- Implemented report generation in the `DataManagerRepository`, including methods to create a run directory and generate comprehensive reports.
- Updated `TrainModelResult` to include `run_id` and `run_dir` attributes for better tracking of training sessions.
2026-04-06 10:38:24 -03:00
vitor-aignosi
18412958d2 refactor: remove MinIO repository references and streamline cleanup process
- Eliminated MinIO repository dependencies from the `Activities`, `Cleanup`, and `Training` classes.
- Updated the `cleanup_resources` method to focus on removing local temporary directories instead of handling MinIO file deletions.
- Adjusted the `TrainModel` class to pass the run directory for cleanup, enhancing resource management in the training workflow.
2026-04-06 09:30:38 -03:00
vitor-aignosi
d2bb588739 Merge branch 'main' into release/SIENTIAPDE-1645 2026-04-06 08:34:15 -03:00
vitor-aignosi
bf2b6b3888 feat: improve error handling and resource cleanup in training workflow
- Updated the `Training` class to raise `ModelTrainingError` on training failures for better error management.
- Enhanced the `run` method in `TrainModel` to return training results and ensure proper resource cleanup, including validation files.
- Refactored exception handling to prevent silent failures during resource cleanup and experiment run updates.
- Adjusted type hints for improved clarity and consistency in method signatures.
2026-04-06 08:26:42 -03:00
github-actions[bot]
dfe05a063a chore: bump version to 1.2.0 [skip ci] 2026-03-30 18:27:46 +00:00
Bruno Domingues
f43fe6a399 Merge pull request #26 from Aignosi/feature/SIENTIAPDE-1717
SIENTIAPDE-1717: Remove MinIO Cleanup Functionality and Streamline Local Temporary File Management
2026-03-30 15:27:20 -03:00
Bruno Domingues
14f91b40aa SIENTIAPDE-1717: Updated documentation to reflect the removal of MinIO data download and cleanup operations from Model Manager workflows. 2026-03-30 14:54:44 -03:00
Bruno Domingues
cce350bfb1 SIENTIAPDE-1717: Remove FILE_DELETE_ERROR from ExperimentStatus enum and related tests. 2026-03-30 14:33:19 -03:00
Bruno Domingues
f5eb8d1e5b SIENTIAPDE-1717: Remove list_bucket_objects method from StorageRepository and its associated test. 2026-03-30 14:18:14 -03:00
Bruno Domingues
7a0961f29d SIENTIAPDE-1717: Remove MinIO cleanup functionality and associated components. This change streamlines the cleanup workflow to focus solely on local temporary directories, removes the ModelTrainingError exception, and updates related configurations, documentation, and tests. 2026-03-30 14:14:10 -03:00
vitor-aignosi
342a02d6f7 feat: enhance training workflow with model metadata loading and refactor data handling
- Introduced a new activity to load model metadata from the model store.
- Refactored training logic to utilize new model metadata and improved parameter handling.
- Updated the `TrainModelParams` class to include additional fields for model configuration.
- Replaced deprecated utility functions with a custom train-test split implementation.
- Removed unused utility functions and cleaned up the data manager repository.
- Adjusted experiment tracking to include model-specific metadata in notifications.
2026-03-24 14:39:28 -03:00
github-actions[bot]
63a94dae0a chore: bump version to 1.1.3 [skip ci] 2026-03-12 17:33:24 +00:00
Bruno Domingues
fb54ee4e9a Merge pull request #25 from Aignosi/fix/SIENTIAPDE-1672
SIENTIAPDE-1672: Streamline Deploy Workflow: Remove Versioning, Pass Branch, Enable VPN
2026-03-12 14:33:02 -03:00
Bruno Domingues
1d096a6dc0 SIENTIAPDE-1672: Streamline deploy workflow by removing dedicated versioning job, passing branch name to reusable deploy, and enabling VPN. 2026-03-12 13:17:07 -03:00
vitor-aignosi
cf5111e520 feat: integrate PluginStore and MinIO repository into model manager activities
- Added PluginStore integration for model management.
- Replaced StorageRepository with MinIORepository in Activities, Cleanup, and Training classes.
- Updated training logic to handle validation files and improved data management.
- Enhanced configuration for MinIO and PluginStore in connectors.
- Removed deprecated model repository and storage repository files.
- Updated environment variable handling for new configurations.
2026-03-11 17:35:05 -03:00
github-actions[bot]
9d71c0cf80 chore: bump version to 1.1.2 [skip ci] 2026-02-20 00:20:15 +00:00
Bruno Domingues
0a0adac86c Merge pull request #24 from Aignosi/fix/SIENTIAPDE-1579
Upgrade MLflow to 2.18.0 and constrain setuptools
2026-02-19 21:19:53 -03:00
Bruno Domingues
41c4490cad SIENTIAPDE-1579: Fix SientiaMlException propagation in ModelServing.search_runs_by_name to correctly raise the exception with its message, resolving a TypeError. Enhance training test script to support scenario-specific CSV files for different test cases. Update search_runs_by_name return type hint and apply minor code formatting. 2026-02-19 21:13:41 -03:00
Bruno Domingues
f952647cf6 SIENTIAPDE-1579: Update MLflow to 2.18.0 and add setuptools version constraint. 2026-02-19 17:56:08 -03:00
github-actions[bot]
b0629cb34a chore: bump version to 1.1.1 [skip ci] 2026-02-19 12:34:59 +00:00
Lucas Kou Kinoshita
8609245e77 Merge pull request #23 from Aignosi/fix/SIENTIAPDE-1579
Fix/SIENTIAPDE-1579: Corrigir bug durante o treinamento de certos arquivos no Soft Sensor
2026-02-19 09:34:34 -03:00
Kou Kinoshita
14f3fbadbe SIENTIAPDE-1579: Switched from replace logic to validation + mapping 2026-02-18 18:57:41 -03:00
Kou Kinoshita
e7e851dbb5 SIENTIAPDE-1579: Lint fix 2026-02-18 18:20:43 -03:00
Kou Kinoshita
5dab1ee576 SIENTIAPDE-1579?: removed duplicated method 2026-02-18 18:18:39 -03:00
Kou Kinoshita
a7922ac3ce SIENTIAPDE-1579: Lint fix 2026-02-18 17:50:58 -03:00
Kou Kinoshita
665870320f SIENTIAPDE-1579: Updated tests and refactored apply filters method 2026-02-18 17:44:46 -03:00
Kou Kinoshita
43cbc31e10 SIENTIAPDE-1579: Lint fixes and formatting 2026-02-18 17:31:58 -03:00
Kou Kinoshita
c6d6f94e05 SIENTIAPDE-1579: Updated tests 2026-02-18 15:05:37 -03:00
Kou Kinoshita
b6eb346a0b SIENTIAPDE-1579: Added test case scenario to test filter, also updated old scenarios 2026-02-18 14:24:17 -03:00
Kou Kinoshita
54a47a1d96 SIENTIAPDE-1579: Added extra test files and support filters implementation 2026-02-18 14:16:21 -03:00
Kou Kinoshita
8b7bd81328 SIENTIAPDE-1579: Fixed data preprocessor 2026-02-18 11:14:27 -03:00
Kou Kinoshita
a4d94dd2ff SIENTIAPDE-1579: Added date format and convert methods 2026-02-13 18:41:46 -03:00
Kou Kinoshita
9e3585afce SIENTIAPDE-1579: Added local and validation flags to test script 2026-02-13 17:51:17 -03:00
github-actions[bot]
f0cdace820 chore: bump version to 1.1.0 [skip ci] 2025-12-19 19:43:10 +00:00
Bruno Domingues
68ed97583a Merge pull request #22 from Aignosi/feature/SIENTIAPDE-1430
Implement Advanced Training Features, Enhance Data Preprocessing, and Bolster Testing & Documentation
2025-12-19 16:42:51 -03:00
Bruno Domingues
df6bf1daba SIENTIAPDE-1430: Refactor static threshold calculation logic and enhance test coverage 2025-12-19 16:19:10 -03:00
Bruno Domingues
06571011f2 SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal.
This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null.
Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
2025-12-19 15:44:18 -03:00
Bruno Domingues
7cbb7022e5 SIENTIAPDE-1430: Refactor DataPreprocessor date filtering, make rce_train radius optional, and introduce constants for model names. 2025-12-19 12:28:07 -03:00
Bruno Domingues
9a77257920 SIENTIAPDE-1430: Update README to document new features, expanded validation, and enhanced testing.
*   Documents support for Polynomial Regression, per-variable lag configuration, date range filtering, and NaN treatment options.
*   Includes details on RCE drift metrics and comprehensive business validation rules for training parameters.
*   Describes new JSON-based integration test scenarios with batch execution and updated 100% code coverage.
*   Refines architecture overview and example workflow parameters.
2025-12-19 11:54:08 -03:00
Bruno Domingues
44dca9b66b SIENTIAPDE-1430: Add option to run all training test scenarios with summary reporting.
- Introduce --all argument to run_training_test.py to execute all available scenarios sequentially.
- Refactor single scenario execution into a run_single_scenario function for better modularity and error handling.
- Implement a print_summary function to report results for multiple scenario runs.
2025-12-18 21:46:51 -03:00
Bruno Domingues
38fb9a9c53 SIENTIAPDE-1430: Remove verbose options and direct print statements from model, preprocessor, and report utilities. 2025-12-18 21:09:12 -03:00
Bruno Domingues
13941fc3f7 SIENTIAPDE-1430: Add unit tests for _configure_datetime_index (handling various datetime column formats), _init_data_preprocessor (with removed intervals), and _extract_model_equation (supporting polynomial features) in TrainingRepository. 2025-12-18 18:58:12 -03:00
Bruno Domingues
71b654f24c SIENTIAPDE-1430: Add test case for ValueError when y_train_pred is None during artifact data initialization. 2025-12-18 18:48:42 -03:00
Bruno Domingues
573834fb99 SIENTIAPDE-1430: Add comprehensive tests for TrainModelParams.validate_business_rules to improve coverage. 2025-12-18 18:46:01 -03:00
Bruno Domingues
70067fd992 SIENTIAPDE-1430: Improve test coverage for LinearRegressionModel and DataPreprocessor. Added extensive new tests covering various methods, edge cases, and scenarios to enhance overall code coverage. 2025-12-18 18:41:03 -03:00
Bruno Domingues
0d50a76f4c SIENTIAPDE-1430: Add unit tests for Silverman radius and RCE drift metrics. 2025-12-18 18:19:35 -03:00
Bruno Domingues
06fd08dc70 SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations. 2025-12-18 17:05:10 -03:00
Bruno Domingues
6e8f87b2a3 SIENTIAPDE-1430: Implement advanced model training capabilities and enhanced data preprocessing. This includes support for Polynomial Regression with configurable degree and interaction terms, flexible per-variable lag configurations, and new data filtering options by date range and removed intervals. Comprehensive business validations are now enforced for all parameters, and MLflow logging has been extended to capture these detailed configurations. Additionally, Reduced Coulomb Energy (RCE) metrics are added for drift detection, with a new changelog documenting all pipeline parameter updates. 2025-12-17 21:39:43 -03:00