526edcb50e
feat: update training workflow and repository management
vitor-aignosi2026-04-09 12:09:52 -03:00
0ae03b246f
feat: update environment configuration and remove deprecated model serving
vitor-aignosi2026-04-07 16:58:50 -03:00
c5cd382350
feat: enhance configuration and scheduling for cleanup processes
vitor-aignosi2026-04-07 12:57:29 -03:00
09ee92f100
feat: enhance configuration and error handling in project setup
vitor-aignosi2026-04-07 10:25:17 -03:00
6b1df7c3a7
feat: enhance training and experiment tracking functionality
vitor-aignosi2026-04-06 15:05:57 -03:00
1352d1ac8f
feat: enhance training model functionality and reporting
vitor-aignosi2026-04-06 11:40:08 -03:00
a8b926649a
feat: enhance training reporting and directory management
vitor-aignosi2026-04-06 10:38:24 -03:00
18412958d2
refactor: remove MinIO repository references and streamline cleanup process
vitor-aignosi2026-04-06 09:30:38 -03:00
d2bb588739
Merge branch 'main' into release/SIENTIAPDE-1645
vitor-aignosi2026-04-06 08:34:15 -03:00
bf2b6b3888
feat: improve error handling and resource cleanup in training workflow
vitor-aignosi2026-04-06 08:26:42 -03:00
dfe05a063a
chore: bump version to 1.2.0 [skip ci]
github-actions[bot]
2026-03-30 18:27:46 +00:00
f43fe6a399
Merge pull request #26 from Aignosi/feature/SIENTIAPDE-1717
Bruno Domingues
2026-03-30 15:27:20 -03:00
14f91b40aa
SIENTIAPDE-1717: Updated documentation to reflect the removal of MinIO data download and cleanup operations from Model Manager workflows.
Bruno Domingues2026-03-30 14:54:44 -03:00
cce350bfb1
SIENTIAPDE-1717: Remove FILE_DELETE_ERROR from ExperimentStatus enum and related tests.
Bruno Domingues2026-03-30 14:33:19 -03:00
f5eb8d1e5b
SIENTIAPDE-1717: Remove list_bucket_objects method from StorageRepository and its associated test.
Bruno Domingues2026-03-30 14:18:14 -03:00
7a0961f29d
SIENTIAPDE-1717: Remove MinIO cleanup functionality and associated components. This change streamlines the cleanup workflow to focus solely on local temporary directories, removes the ModelTrainingError exception, and updates related configurations, documentation, and tests.
Bruno Domingues2026-03-30 14:14:10 -03:00
342a02d6f7
feat: enhance training workflow with model metadata loading and refactor data handling
vitor-aignosi2026-03-24 14:39:28 -03:00
63a94dae0a
chore: bump version to 1.1.3 [skip ci]
github-actions[bot]
2026-03-12 17:33:24 +00:00
fb54ee4e9a
Merge pull request #25 from Aignosi/fix/SIENTIAPDE-1672
Bruno Domingues
2026-03-12 14:33:02 -03:00
1d096a6dc0
SIENTIAPDE-1672: Streamline deploy workflow by removing dedicated versioning job, passing branch name to reusable deploy, and enabling VPN.
Bruno Domingues2026-03-12 13:17:07 -03:00
cf5111e520
feat: integrate PluginStore and MinIO repository into model manager activities
vitor-aignosi2026-03-11 17:35:05 -03:00
9d71c0cf80
chore: bump version to 1.1.2 [skip ci]
github-actions[bot]
2026-02-20 00:20:15 +00:00
0a0adac86c
Merge pull request #24 from Aignosi/fix/SIENTIAPDE-1579
Bruno Domingues
2026-02-19 21:19:53 -03:00
41c4490cad
SIENTIAPDE-1579: Fix SientiaMlException propagation in ModelServing.search_runs_by_name to correctly raise the exception with its message, resolving a TypeError. Enhance training test script to support scenario-specific CSV files for different test cases. Update search_runs_by_name return type hint and apply minor code formatting.
Bruno Domingues2026-02-19 21:13:41 -03:00
f952647cf6
SIENTIAPDE-1579: Update MLflow to 2.18.0 and add setuptools version constraint.
Bruno Domingues2026-02-19 17:56:08 -03:00
b0629cb34a
chore: bump version to 1.1.1 [skip ci]
github-actions[bot]
2026-02-19 12:34:59 +00:00
8609245e77
Merge pull request #23 from Aignosi/fix/SIENTIAPDE-1579
Lucas Kou Kinoshita
2026-02-19 09:34:34 -03:00
14f3fbadbe
SIENTIAPDE-1579: Switched from replace logic to validation + mapping
Kou Kinoshita
2026-02-18 18:57:41 -03:00
b6eb346a0b
SIENTIAPDE-1579: Added test case scenario to test filter, also updated old scenarios
Kou Kinoshita
2026-02-18 14:24:17 -03:00
54a47a1d96
SIENTIAPDE-1579: Added extra test files and support filters implementation
Kou Kinoshita
2026-02-18 14:16:21 -03:00
8b7bd81328
SIENTIAPDE-1579: Fixed data preprocessor
Kou Kinoshita
2026-02-18 11:14:27 -03:00
a4d94dd2ff
SIENTIAPDE-1579: Added date format and convert methods
Kou Kinoshita
2026-02-13 18:41:46 -03:00
9e3585afce
SIENTIAPDE-1579: Added local and validation flags to test script
Kou Kinoshita
2026-02-13 17:51:17 -03:00
f0cdace820
chore: bump version to 1.1.0 [skip ci]
github-actions[bot]
2025-12-19 19:43:10 +00:00
68ed97583a
Merge pull request #22 from Aignosi/feature/SIENTIAPDE-1430
Bruno Domingues
2025-12-19 16:42:51 -03:00
df6bf1daba
SIENTIAPDE-1430: Refactor static threshold calculation logic and enhance test coverage
Bruno Domingues2025-12-19 16:19:10 -03:00
06571011f2
SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal. This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null. Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
Bruno Domingues2025-12-19 15:44:18 -03:00
7cbb7022e5
SIENTIAPDE-1430: Refactor DataPreprocessor date filtering, make rce_train radius optional, and introduce constants for model names.
Bruno Domingues2025-12-19 12:28:07 -03:00
9a77257920
SIENTIAPDE-1430: Update README to document new features, expanded validation, and enhanced testing. * Documents support for Polynomial Regression, per-variable lag configuration, date range filtering, and NaN treatment options. * Includes details on RCE drift metrics and comprehensive business validation rules for training parameters. * Describes new JSON-based integration test scenarios with batch execution and updated 100% code coverage. * Refines architecture overview and example workflow parameters.
Bruno Domingues2025-12-19 11:54:08 -03:00
44dca9b66b
SIENTIAPDE-1430: Add option to run all training test scenarios with summary reporting. - Introduce --all argument to run_training_test.py to execute all available scenarios sequentially. - Refactor single scenario execution into a run_single_scenario function for better modularity and error handling. - Implement a print_summary function to report results for multiple scenario runs.
Bruno Domingues2025-12-18 21:46:51 -03:00
38fb9a9c53
SIENTIAPDE-1430: Remove verbose options and direct print statements from model, preprocessor, and report utilities.
Bruno Domingues2025-12-18 21:09:12 -03:00
13941fc3f7
SIENTIAPDE-1430: Add unit tests for _configure_datetime_index (handling various datetime column formats), _init_data_preprocessor (with removed intervals), and _extract_model_equation (supporting polynomial features) in TrainingRepository.
Bruno Domingues2025-12-18 18:58:12 -03:00
71b654f24c
SIENTIAPDE-1430: Add test case for ValueError when y_train_pred is None during artifact data initialization.
Bruno Domingues2025-12-18 18:48:42 -03:00
573834fb99
SIENTIAPDE-1430: Add comprehensive tests for TrainModelParams.validate_business_rules to improve coverage.
Bruno Domingues2025-12-18 18:46:01 -03:00
70067fd992
SIENTIAPDE-1430: Improve test coverage for LinearRegressionModel and DataPreprocessor. Added extensive new tests covering various methods, edge cases, and scenarios to enhance overall code coverage.
Bruno Domingues2025-12-18 18:41:03 -03:00
0d50a76f4c
SIENTIAPDE-1430: Add unit tests for Silverman radius and RCE drift metrics.
Bruno Domingues2025-12-18 18:19:35 -03:00
06fd08dc70
SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations.
Bruno Domingues2025-12-18 17:05:10 -03:00
6e8f87b2a3
SIENTIAPDE-1430: Implement advanced model training capabilities and enhanced data preprocessing. This includes support for Polynomial Regression with configurable degree and interaction terms, flexible per-variable lag configurations, and new data filtering options by date range and removed intervals. Comprehensive business validations are now enforced for all parameters, and MLflow logging has been extended to capture these detailed configurations. Additionally, Reduced Coulomb Energy (RCE) metrics are added for drift detection, with a new changelog documenting all pipeline parameter updates.
Bruno Domingues2025-12-17 21:39:43 -03:00
f05b4d3253
Merge pull request #19 from Aignosi/fix/SIENTIAPDE-1352
Bruno Domingues
2025-12-04 18:12:19 -03:00
8cb39605d2
SIENTIAPDE-1352: Configure helm deployment to update version in pyproject and values files.
Bruno Domingues2025-12-04 18:08:44 -03:00
0a3135a8a5
Merge pull request #18 from Aignosi/release/SIENTIAPDE-1352
Bruno Domingues
2025-12-04 16:44:14 -03:00
a974926f3d
SIENTIAPDE-1352: Simplify deploy workflow by inheriting secrets.
Bruno Domingues2025-12-04 16:41:03 -03:00
463d086c5e
Merge branch 'main' into release/SIENTIAPDE-1352
Bruno Domingues2025-12-04 16:36:25 -03:00
e337373f59
Merge pull request #17 from Aignosi/release/SIENTIAPDE-1352
Bruno Domingues
2025-12-04 13:54:10 -03:00
5fb8aead22
SIENTIAPDE-1352: Update deploy workflow to use reusable workflow and add permissions, update README with CI/CD details.
Bruno Domingues2025-12-04 12:54:03 -03:00
f0aaa6e2c1
feat(deploy): add write-all permissions to deploy workflow
Bruno Domingues2025-12-04 12:48:11 -03:00
9871867b76
Merge pull request #16 from Aignosi/release/SIENTIAPDE-1352
Bruno Domingues
2025-12-04 12:41:54 -03:00
cdaafd12cc
SIENTIAPDE-1352: Add deploy workflow and update README with local repository execution instructions.
Bruno Domingues2025-12-04 12:09:50 -03:00
46df241f7d
SIENTIAPDE-1352: Add documentation for local GitHub Actions testing with act.
Bruno Domingues2025-12-02 17:28:04 -03:00
2b729c65e6
SIENTIAPDE-1352: Refactor quality gate workflow to use reusable dataops-module-quality-gate workflow and update .gitignore. ( -230, +19 lines)
Bruno Domingues2025-12-02 17:20:31 -03:00
3a61633ec6
Merge pull request #15 from Aignosi/feature/SIENTIAPDE-1350
Bruno Domingues
2025-11-27 12:25:58 -03:00
9d3ecc1c2f
SIENTIAPDE-1350: Simplify exception handling for file and directory deletion in cleanup activity
Bruno Domingues2025-11-27 11:00:16 -03:00
486e1756c9
SIENTIAPDE-1350: Implement automated file cleanup workflow and update documentation. Added a new workflow for automated cleanup of stale files from MinIO and local filesystem, including scheduled execution and configurable retention. Updated the README with detailed information about the new workflow, its configuration, and related metrics. (118 additions, 5 deletions)
Bruno Domingues2025-11-27 10:22:38 -03:00
5dcb62ef4b
SIENTIAPDE-1350: Integrate cleanup workflow and update default task queue names.
Bruno Domingues2025-11-26 22:11:20 -03:00
11aaa0780f
SIENTIAPDE-1350: Improve Docker image build context and fix queue names
Bruno Domingues2025-11-26 17:52:44 -03:00
c281969381
SIENTIAPDE-1350: Add test to ensure worker starts despite cleanup schedule creation failure.
Bruno Domingues2025-11-26 11:30:22 -03:00
bc917e97d8
SIENTIAPDE-1350: Add cleanup schedule tests and configure worker with task queues from env vars
Bruno Domingues2025-11-25 19:21:55 -03:00
b028dc1b7d
SIENTIAPDE-1350: Implement scheduled cleanup workflow using Temporal schedules. Adds schedule creation to worker startup and configures environment variables.
Bruno Domingues2025-11-25 18:12:53 -03:00
9ecac9c8fd
SIENTIAPDE-1350: Update values.yaml with new environment variables and remove outdated task from todo-list.txt
Bruno Domingues2025-11-25 17:03:06 -03:00
823df2d65d
SIENTIAPDE-1350: Refactor: Replace AsyncMock with MagicMock in cleanup tests for synchronous calls
Bruno Domingues2025-11-25 16:54:45 -03:00
33480fa108
SIENTIAPDE-1350: Add unit tests for the CleanupFiles workflow, covering both input bucket and default bucket scenarios.
Bruno Domingues2025-11-25 16:44:51 -03:00
d0b5b74f88
SIENTIAPDE-1350: Add tests for close method and paginated list_bucket_objects
Bruno Domingues2025-11-25 16:04:59 -03:00
d2c4803a75
SIENTIAPDE-1350: Add unit tests for the Cleanup activity, ensuring 100% code coverage, and update todo-list.
Bruno Domingues2025-11-25 16:01:10 -03:00
f3c88885ea
SIENTIAPDE-1350: Configure cleanup test script and improve test isolation. This commit configures the cleanup test script to load environment variables from a .env file and use them for Temporal connection. It also adds a fixture to clean up temporary directories created by tests, ensuring better test isolation and preventing potential conflicts. Additionally, it adds the 'uri' property to the MongoDB config.
Bruno Domingues2025-11-25 00:44:55 -03:00
463906073e
SIENTIAPDE-1350: Refactor: Improve logging, configuration, and resource management. Includes .env updates, client initialization logging, and resource closing.
Bruno Domingues2025-11-24 23:08:29 -03:00
2ee66552d2
SIENTIAPDE-1350: Configure separate task queues for train and cleanup workers, update sientia-dataops-library to 1.6.1 and refactor prometheus server startup to use logger.
Bruno Domingues2025-11-24 11:21:43 -03:00
53f49d7a97
SIENTIAPDE-1350: Implement file cleanup workflow and activities, including MinIO and local directory cleanup, configuration, and metrics.
Bruno Domingues2025-11-19 18:38:00 -03:00
cea3ef60cb
Merge pull request #14 from Aignosi/feature/SIENTIAPDE-1349
Bruno Domingues
2025-11-11 11:46:49 -03:00
731a23a92e
SIENTIAPDE-1349: Add Grafana dashboard configuration to values.yaml
Bruno Domingues2025-11-11 11:37:28 -03:00
e51daca073
Merge pull request #13 from Aignosi/feature/SIENTIAPDE-1309
Bruno Domingues
2025-11-07 16:32:36 -03:00
f232406bd3
SIENTIAPDE-1309: Improve .gitignore to preserve .gitkeep in model_manager reports temp directory.
Bruno Domingues2025-11-07 15:25:10 -03:00
18103f2436
SIENTIAPDE-1309: Refine .gitignore to ignore test run reports within the model_manager/reports/temp directory, while preserving the temp directory itself.
Bruno Domingues2025-11-07 15:19:21 -03:00
7ff54a1112
SIENTIAPDE-1309: Improve Dockerignore, Gitignore, Dockerfile, README, model repository, tests, todo list and values.yaml files.
Bruno Domingues2025-11-07 15:08:50 -03:00
258bad84c6
SIENTIAPDE-1309: Add ignore rule for test run reports in model_manager/reports
Bruno Domingues2025-11-06 15:59:44 -03:00
vitor-aignosi
github-actions[bot]
Bruno Domingues