Kou Kinoshita
e7e851dbb5
SIENTIAPDE-1579: Lint fix
2026-02-18 18:20:43 -03:00
Kou Kinoshita
5dab1ee576
SIENTIAPDE-1579?: removed duplicated method
2026-02-18 18:18:39 -03:00
Kou Kinoshita
665870320f
SIENTIAPDE-1579: Updated tests and refactored apply filters method
2026-02-18 17:44:46 -03:00
Kou Kinoshita
43cbc31e10
SIENTIAPDE-1579: Lint fixes and formatting
2026-02-18 17:31:58 -03:00
Kou Kinoshita
54a47a1d96
SIENTIAPDE-1579: Added extra test files and support filters implementation
2026-02-18 14:16:21 -03:00
Kou Kinoshita
8b7bd81328
SIENTIAPDE-1579: Fixed data preprocessor
2026-02-18 11:14:27 -03:00
Kou Kinoshita
a4d94dd2ff
SIENTIAPDE-1579: Added date format and convert methods
2026-02-13 18:41:46 -03:00
Bruno Domingues
df6bf1daba
SIENTIAPDE-1430: Refactor static threshold calculation logic and enhance test coverage
2025-12-19 16:19:10 -03:00
Bruno Domingues
06571011f2
SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal.
...
This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null.
Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
2025-12-19 15:44:18 -03:00
Bruno Domingues
7cbb7022e5
SIENTIAPDE-1430: Refactor DataPreprocessor date filtering, make rce_train radius optional, and introduce constants for model names.
2025-12-19 12:28:07 -03:00
Bruno Domingues
38fb9a9c53
SIENTIAPDE-1430: Remove verbose options and direct print statements from model, preprocessor, and report utilities.
2025-12-18 21:09:12 -03:00
Bruno Domingues
70067fd992
SIENTIAPDE-1430: Improve test coverage for LinearRegressionModel and DataPreprocessor. Added extensive new tests covering various methods, edge cases, and scenarios to enhance overall code coverage.
2025-12-18 18:41:03 -03:00
Bruno Domingues
06fd08dc70
SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations.
2025-12-18 17:05:10 -03:00
Bruno Domingues
6e8f87b2a3
SIENTIAPDE-1430: Implement advanced model training capabilities and enhanced data preprocessing. This includes support for Polynomial Regression with configurable degree and interaction terms, flexible per-variable lag configurations, and new data filtering options by date range and removed intervals. Comprehensive business validations are now enforced for all parameters, and MLflow logging has been extended to capture these detailed configurations. Additionally, Reduced Coulomb Energy (RCE) metrics are added for drift detection, with a new changelog documenting all pipeline parameter updates.
2025-12-17 21:39:43 -03:00
Bruno Domingues
9d3ecc1c2f
SIENTIAPDE-1350: Simplify exception handling for file and directory deletion in cleanup activity
2025-11-27 11:00:16 -03:00
Bruno Domingues
5dcb62ef4b
SIENTIAPDE-1350: Integrate cleanup workflow and update default task queue names.
2025-11-26 22:11:20 -03:00
Bruno Domingues
b028dc1b7d
SIENTIAPDE-1350: Implement scheduled cleanup workflow using Temporal schedules. Adds schedule creation to worker startup and configures environment variables.
2025-11-25 18:12:53 -03:00
Bruno Domingues
463906073e
SIENTIAPDE-1350: Refactor: Improve logging, configuration, and resource management. Includes .env updates, client initialization logging, and resource closing.
2025-11-24 23:08:29 -03:00
Bruno Domingues
2ee66552d2
SIENTIAPDE-1350: Configure separate task queues for train and cleanup workers, update sientia-dataops-library to 1.6.1 and refactor prometheus server startup to use logger.
2025-11-24 11:21:43 -03:00
Bruno Domingues
53f49d7a97
SIENTIAPDE-1350: Implement file cleanup workflow and activities, including MinIO and local directory cleanup, configuration, and metrics.
2025-11-19 18:38:00 -03:00
Bruno Domingues
f232406bd3
SIENTIAPDE-1309: Improve .gitignore to preserve .gitkeep in model_manager reports temp directory.
2025-11-07 15:25:10 -03:00
Bruno Domingues
7ff54a1112
SIENTIAPDE-1309: Improve Dockerignore, Gitignore, Dockerfile, README, model repository, tests, todo list and values.yaml files.
2025-11-07 15:08:50 -03:00
Bruno Domingues
c7f44a2423
SIENTIAPDE-1309: Update README with Helm instructions and refactor experiment status messages. Also, update values.yaml with new image and configurations.
2025-11-06 15:34:03 -03:00
Bruno Domingues
9f56a128e6
SIENTIAPDE-1308: Update documentation, remove install_dependencies.sh script from README, correct Training class initialization, and update sientia-dataops-library version in requirements.txt.
2025-11-03 17:35:33 -03:00
Bruno Domingues
f32d647d1c
SIENTIAPDE-1307: Implement metrics for training activities and workflow executions, tracking success/failure status.
2025-11-03 14:41:37 -03:00
Bruno Domingues
db581f60f9
SIENTIAPDE-1307: Integrate metrics controller and update sientia-dataops-library to 1.5.1. This change adds metrics collection capabilities to activities and updates the dataops library dependency. (18 files changed, 143 insertions(+), 16 deletions(-))
2025-10-31 17:06:26 -03:00
Bruno Domingues
bb25a19d8c
SIENTIAPDE-1241: Simplify worker shutdown logic by removing conditional checks.
2025-10-30 15:14:43 -03:00
Kou-Kinoshita
26fca36d82
SIENTIAPDE-1241: Refactored method into smaller ones
2025-10-29 16:45:30 -03:00
Kou-Kinoshita
25ad4729f7
SIENTIAPDE-1321: Formatted files
2025-10-29 10:24:25 -03:00
Kou-Kinoshita
a80d65d7ba
SIENTIAPDE-1321: Added equation related methods
2025-10-24 08:03:27 -03:00
Bruno Domingues
f07bc8ff30
SIENTIAPDE-1241: Refactor: Remove redundant logging and fix duplicate logs
...
This commit removes redundant logging statements from training and experiment tracking activities, preventing duplicate log entries. It also introduces a logger helper to disable log propagation, further addressing the duplicate logs issue. Additionally, the Makefile, run_coverage.sh, setup_port_forwards.sh, and simulator/Dockerfile files were removed as they are no longer needed.
2025-10-22 22:39:26 -03:00
Bruno Domingues
ac97092ebf
SIENTIAPDE-1241: Remove unused metadata fields and update todo list.
2025-10-22 21:51:29 -03:00
Bruno Domingues
94697215aa
SIENTIAPDE-1241: Refactor: Improve documentation, exception handling, and configuration in model manager. This commit enhances clarity and robustness by adding detailed docstrings to methods, standardizing exception handling with custom types, and simplifying MLflow configuration.
2025-10-22 21:48:12 -03:00
Bruno Domingues
758ffb10b6
SIENTIAPDE-1241: Fix: Correctly handle feature normalization and denormalization, and add scaler parameters to metadata
2025-10-22 21:26:11 -03:00
Bruno Domingues
42a7e82c82
SIENTIAPDE-1241: Suppress sklearn FutureWarning regarding 'squared' deprecation.
2025-10-22 18:28:13 -03:00
Bruno Domingues
5789a13023
SIENTIAPDE-1241: refactor train_model workflow due to I/O errors.
2025-10-22 15:37:56 -03:00
Bruno Domingues
ab7c93480b
SIENTIAPDE-1255: Refactor: Move extra pip requirements to environment variable and use constants for preprocessor steps.
2025-10-21 09:20:23 -03:00
Bruno Domingues
4a32d712ba
SIENTIAPDE-1255: Implement object destructors for Activities and ExperimentTracking to prevent AttributeError during garbage collection, and mock workflow logger in tests to avoid RuntimeWarning.
2025-10-20 17:53:49 -03:00
Bruno Domingues
52be5c4e8e
SIENTIAPDE-1255: Improve documentation for split_train_test function in utils.py
2025-10-20 17:41:05 -03:00
Bruno Domingues
0dbe179367
SIENTIAPDE-1255: Add thread-safety documentation and I/O notes to Reports class, improve save_all_sections documentation and error handling.
2025-10-20 17:38:44 -03:00
Bruno Domingues
2292ae57cc
SIENTIAPDE-1255: Add thread-safety documentation and in-place modification warnings to LinearRegressionModel and DataPreprocessor classes.
2025-10-20 17:27:58 -03:00
Bruno Domingues
56f3db350c
SIENTIAPDE-1255: Refactor ModelServing class to improve thread safety, resource management, and documentation. Adds context manager for experiment saving and clarifies thread-safety concerns.
2025-10-20 17:16:45 -03:00
Bruno Domingues
56a21a16da
SIENTIAPDE-1255: Integrate sientia-mlops-library into model-manager, adding model serving, reporting, and updated model definitions.
2025-10-20 16:55:52 -03:00
Bruno Domingues
9e31ab679b
SIENTIAPDE-1255: Implement MLFlow artifact management and model persistence
...
This commit introduces a new model_repository.py to handle MLFlow artifact generation and model persistence. It also updates the README to reflect this change and modifies training_repository.py to separate training and MLFlow operations.
2025-10-17 09:11:54 -03:00
Bruno Domingues
a66b996cc1
SIENTIAPDE-1255: Refactor MLFlow activities for training operations and update metrics
...
This commit refactors the MLFlow activities to focus on model training rather than prediction operations. It removes prediction-related activities and metrics, and updates the MLFlow activity descriptions to reflect the change in focus. The README is also updated to reflect these changes.
2025-10-17 00:51:29 -03:00
Bruno Domingues
85a8ba1e68
SIENTIAPDE-1255: Refactor data quality gates to training focused metrics and repositories. This commit removes the data quality gates and filters, focusing on training-specific metrics and data repositories. It also updates the README to reflect these changes, including new training metrics and a streamlined data services section.
2025-10-17 00:39:06 -03:00
Bruno Domingues
e191e7849f
SIENTIAPDE-1255: Refactor Model Manager to focus on ML model training pipeline and simplify architecture. This includes removing prediction workflows, updating core functionality, and improving parameter validation and resource management.
...
README.md | 411 deletions(-) 117 insertions(+)
1 file changed, 117 insertions(+), 411 deletions(-)
2025-10-16 18:26:56 -03:00
Bruno Domingues
7abd951806
SIENTIAPDE-1255: Refactor worker to support only the train_model-queue and remove prediction workflows.
2025-10-16 18:15:52 -03:00
Bruno Domingues
98c216733b
SIENTIAPDE-1253: Enhance error logging in train_model workflow with rich context for debugging.
2025-10-15 16:19:15 -03:00
Bruno Domingues
305a39b44c
SIENTIAPDE-1253: Implement business rule validation for training parameters. Adds a validate_business_rules method to the TrainModelParams class to enforce constraints on training parameters, improving data integrity and preventing errors. Also updates tests to include target variable in variable columns and adds tests for business rule validations.
2025-10-15 16:16:09 -03:00
