Bruno Domingues
06571011f2
SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal.
...
This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null.
Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
2025-12-19 15:44:18 -03:00
Bruno Domingues
7cbb7022e5
SIENTIAPDE-1430: Refactor DataPreprocessor date filtering, make rce_train radius optional, and introduce constants for model names.
2025-12-19 12:28:07 -03:00
Bruno Domingues
9a77257920
SIENTIAPDE-1430: Update README to document new features, expanded validation, and enhanced testing.
...
* Documents support for Polynomial Regression, per-variable lag configuration, date range filtering, and NaN treatment options.
* Includes details on RCE drift metrics and comprehensive business validation rules for training parameters.
* Describes new JSON-based integration test scenarios with batch execution and updated 100% code coverage.
* Refines architecture overview and example workflow parameters.
2025-12-19 11:54:08 -03:00
Bruno Domingues
44dca9b66b
SIENTIAPDE-1430: Add option to run all training test scenarios with summary reporting.
...
- Introduce --all argument to run_training_test.py to execute all available scenarios sequentially.
- Refactor single scenario execution into a run_single_scenario function for better modularity and error handling.
- Implement a print_summary function to report results for multiple scenario runs.
2025-12-18 21:46:51 -03:00
Bruno Domingues
38fb9a9c53
SIENTIAPDE-1430: Remove verbose options and direct print statements from model, preprocessor, and report utilities.
2025-12-18 21:09:12 -03:00
Bruno Domingues
13941fc3f7
SIENTIAPDE-1430: Add unit tests for _configure_datetime_index (handling various datetime column formats), _init_data_preprocessor (with removed intervals), and _extract_model_equation (supporting polynomial features) in TrainingRepository.
2025-12-18 18:58:12 -03:00
Bruno Domingues
71b654f24c
SIENTIAPDE-1430: Add test case for ValueError when y_train_pred is None during artifact data initialization.
2025-12-18 18:48:42 -03:00
Bruno Domingues
573834fb99
SIENTIAPDE-1430: Add comprehensive tests for TrainModelParams.validate_business_rules to improve coverage.
2025-12-18 18:46:01 -03:00
Bruno Domingues
70067fd992
SIENTIAPDE-1430: Improve test coverage for LinearRegressionModel and DataPreprocessor. Added extensive new tests covering various methods, edge cases, and scenarios to enhance overall code coverage.
2025-12-18 18:41:03 -03:00
Bruno Domingues
0d50a76f4c
SIENTIAPDE-1430: Add unit tests for Silverman radius and RCE drift metrics.
2025-12-18 18:19:35 -03:00
Bruno Domingues
06fd08dc70
SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations.
2025-12-18 17:05:10 -03:00
Bruno Domingues
6e8f87b2a3
SIENTIAPDE-1430: Implement advanced model training capabilities and enhanced data preprocessing. This includes support for Polynomial Regression with configurable degree and interaction terms, flexible per-variable lag configurations, and new data filtering options by date range and removed intervals. Comprehensive business validations are now enforced for all parameters, and MLflow logging has been extended to capture these detailed configurations. Additionally, Reduced Coulomb Energy (RCE) metrics are added for drift detection, with a new changelog documenting all pipeline parameter updates.
2025-12-17 21:39:43 -03:00
Bruno Domingues
4d6674758c
SIENTIAPDE-1430: Remove todo-list.txt
2025-12-17 15:45:00 -03:00
Bruno Domingues
cf73def257
ci(workflows): Trigger PR workflows on release and feature branches
2025-12-09 11:54:06 -03:00
github-actions[bot]
923f2cb675
chore: bump version to 1.0.3 [skip ci]
2025-12-05 14:45:10 +00:00
Bruno Domingues
c39a1428c7
Merge pull request #21 from Aignosi/fix/SIENTIAPDE-1352
...
SIENTIAPDE-1352: Update workflow references to main and fix quality gate
2025-12-05 11:41:02 -03:00
Bruno Domingues
9d118fa6fa
SIENTIAPDE-1352: Update workflow references to use main branch and correct quality gate workflow.
2025-12-05 11:32:59 -03:00
github-actions[bot]
530e8cbb08
chore: bump version to 1.0.2 [skip ci]
2025-12-04 22:39:52 +00:00
Bruno Domingues
6d19acd5a2
Merge pull request #20 from Aignosi/fix/SIENTIAPDE-1352
...
Update README with comprehensive project documentation
2025-12-04 19:36:06 -03:00
Bruno Domingues
32aa66e5d8
SIENTIAPDE-1352: Update README with improved instructions, project structure details, and environment variable descriptions.
2025-12-04 19:31:35 -03:00
Bruno Domingues
01b8c06d75
Merge branch 'main' into fix/SIENTIAPDE-1352
2025-12-04 19:26:08 -03:00
Bruno Domingues
f05b4d3253
Merge pull request #19 from Aignosi/fix/SIENTIAPDE-1352
...
SIENTIAPDE-1352: Configure Helm deployment to update version in pyproject and values files
2025-12-04 18:12:19 -03:00
Bruno Domingues
8cb39605d2
SIENTIAPDE-1352: Configure helm deployment to update version in pyproject and values files.
2025-12-04 18:08:44 -03:00
Bruno Domingues
0a3135a8a5
Merge pull request #18 from Aignosi/release/SIENTIAPDE-1352
...
Simplify deploy workflow by inheriting secrets
2025-12-04 16:44:14 -03:00
Bruno Domingues
a974926f3d
SIENTIAPDE-1352: Simplify deploy workflow by inheriting secrets.
2025-12-04 16:41:03 -03:00
Bruno Domingues
463d086c5e
Merge branch 'main' into release/SIENTIAPDE-1352
2025-12-04 16:36:25 -03:00
Bruno Domingues
e337373f59
Merge pull request #17 from Aignosi/release/SIENTIAPDE-1352
...
SIENTIAPDE-1352: Update deploy workflow to use reusable workflow and add permissions, update README with CI/CD details.
2025-12-04 13:54:10 -03:00
Bruno Domingues
5fb8aead22
SIENTIAPDE-1352: Update deploy workflow to use reusable workflow and add permissions, update README with CI/CD details.
2025-12-04 12:54:03 -03:00
Bruno Domingues
f0aaa6e2c1
feat(deploy): add write-all permissions to deploy workflow
...
This commit adds write-all permissions to the get-version and deploy jobs in the deploy workflow. This allows the workflow to properly interact with the repository for versioning and deployment tasks.
2025-12-04 12:48:11 -03:00
Bruno Domingues
9871867b76
Merge pull request #16 from Aignosi/release/SIENTIAPDE-1352
...
SIENTIAPDE-1352: Enhance CI/CD and Local Development Setup
2025-12-04 12:41:54 -03:00
Bruno Domingues
cdaafd12cc
SIENTIAPDE-1352: Add deploy workflow and update README with local repository execution instructions.
2025-12-04 12:09:50 -03:00
Bruno Domingues
46df241f7d
SIENTIAPDE-1352: Add documentation for local GitHub Actions testing with act.
...
This commit adds a new section to the README.md file detailing how to use the 'act' tool for local testing of GitHub Actions workflows. It includes installation instructions, configuration details, usage examples, and troubleshooting tips.
2025-12-02 17:28:04 -03:00
Bruno Domingues
2b729c65e6
SIENTIAPDE-1352: Refactor quality gate workflow to use reusable dataops-module-quality-gate workflow and update .gitignore. ( -230, +19 lines)
2025-12-02 17:20:31 -03:00
Bruno Domingues
3a61633ec6
Merge pull request #15 from Aignosi/feature/SIENTIAPDE-1350
...
SIENTIAPDE-1350: Implement Automated File Cleanup Workflow with Temporal Schedules
2025-11-27 12:25:58 -03:00
Bruno Domingues
9d3ecc1c2f
SIENTIAPDE-1350: Simplify exception handling for file and directory deletion in cleanup activity
2025-11-27 11:00:16 -03:00
Bruno Domingues
486e1756c9
SIENTIAPDE-1350: Implement automated file cleanup workflow and update documentation. Added a new workflow for automated cleanup of stale files from MinIO and local filesystem, including scheduled execution and configurable retention. Updated the README with detailed information about the new workflow, its configuration, and related metrics. (118 additions, 5 deletions)
2025-11-27 10:22:38 -03:00
Bruno Domingues
5dcb62ef4b
SIENTIAPDE-1350: Integrate cleanup workflow and update default task queue names.
2025-11-26 22:11:20 -03:00
Bruno Domingues
11aaa0780f
SIENTIAPDE-1350: Improve Docker image build context and fix queue names
...
This commit refactors the .dockerignore file to use a whitelist approach, significantly reducing the Docker image size by excluding unnecessary files. It also fixes the queue names in values.yaml and adds TEMPORAL_USE_TLS to the test environment.
2025-11-26 17:52:44 -03:00
Bruno Domingues
c281969381
SIENTIAPDE-1350: Add test to ensure worker starts despite cleanup schedule creation failure.
2025-11-26 11:30:22 -03:00
Bruno Domingues
bc917e97d8
SIENTIAPDE-1350: Add cleanup schedule tests and configure worker with task queues from env vars
2025-11-25 19:21:55 -03:00
Bruno Domingues
b028dc1b7d
SIENTIAPDE-1350: Implement scheduled cleanup workflow using Temporal schedules. Adds schedule creation to worker startup and configures environment variables.
2025-11-25 18:12:53 -03:00
Bruno Domingues
9ecac9c8fd
SIENTIAPDE-1350: Update values.yaml with new environment variables and remove outdated task from todo-list.txt
2025-11-25 17:03:06 -03:00
Bruno Domingues
823df2d65d
SIENTIAPDE-1350: Refactor: Replace AsyncMock with MagicMock in cleanup tests for synchronous calls
2025-11-25 16:54:45 -03:00
Bruno Domingues
33480fa108
SIENTIAPDE-1350: Add unit tests for the CleanupFiles workflow, covering both input bucket and default bucket scenarios.
2025-11-25 16:44:51 -03:00
Bruno Domingues
d0b5b74f88
SIENTIAPDE-1350: Add tests for close method and paginated list_bucket_objects
2025-11-25 16:04:59 -03:00
Bruno Domingues
d2c4803a75
SIENTIAPDE-1350: Add unit tests for the Cleanup activity, ensuring 100% code coverage, and update todo-list.
2025-11-25 16:01:10 -03:00
Bruno Domingues
f3c88885ea
SIENTIAPDE-1350: Configure cleanup test script and improve test isolation. This commit configures the cleanup test script to load environment variables from a .env file and use them for Temporal connection. It also adds a fixture to clean up temporary directories created by tests, ensuring better test isolation and preventing potential conflicts. Additionally, it adds the 'uri' property to the MongoDB config.
2025-11-25 00:44:55 -03:00
Bruno Domingues
463906073e
SIENTIAPDE-1350: Refactor: Improve logging, configuration, and resource management. Includes .env updates, client initialization logging, and resource closing.
2025-11-24 23:08:29 -03:00
Bruno Domingues
2ee66552d2
SIENTIAPDE-1350: Configure separate task queues for train and cleanup workers, update sientia-dataops-library to 1.6.1 and refactor prometheus server startup to use logger.
2025-11-24 11:21:43 -03:00
Bruno Domingues
53f49d7a97
SIENTIAPDE-1350: Implement file cleanup workflow and activities, including MinIO and local directory cleanup, configuration, and metrics.
2025-11-19 18:38:00 -03:00
