Commit Graph

236 Commits

Author SHA1 Message Date
vitor-aignosi
0d00a8c3d2 chore: update evidently dependency to version 0.7.21 in requirements files 2026-04-16 10:59:10 -03:00
vitor-aignosi
8d4dc65aaa feat: enhance report generation with target name and content injection
- Added target_name parameter to Reports class for improved report context.
- Updated inject_content function to ensure proper handling of HTML sections.
- Modified DataManagerRepository to join predictions with training and validation data for accurate report generation.
2026-04-16 10:26:42 -03:00
vitor-aignosi
1e05ebe147 feat: add experiment_name to TrainModelResult and update training logic
- Introduced experiment_name parameter in TrainModelResult to enhance tracking of training experiments.
- Updated the Training class to utilize run_name and experiment_name for improved MLflow run management.
2026-04-16 09:43:39 -03:00
vitor-aignosi
c75a5921f3 feat: add run_name to TrainModelResult for enhanced tracking
- Introduced run_name parameter in TrainModelResult to improve identification of training runs based on model type, name, and experiment run ID.
2026-04-16 09:34:36 -03:00
vitor-aignosi
e658c3036f feat: add timezone handling for DataFrame index
- Implemented _set_timezone_on_index method to ensure DataFrame indices are set to UTC if not already timezone-aware.
- Updated training and validation DataFrame processing to include timezone configuration for improved data consistency.
2026-04-16 09:20:04 -03:00
vitor-aignosi
409cad23c0 SIENTIAPDE-1645: Update sientia_model dependency to 0.8.1 2026-04-15 17:05:16 -03:00
vitor-aignosi
a83cdbfde0 chore: ensure consistent formatting in quality-gate.yml
- Maintained existing configuration for secrets inheritance in the quality-gate.yml file.
- Added a newline at the end of the file for consistency with coding standards.
2026-04-14 13:44:34 -03:00
vitor-aignosi
3bee743cfd feat: enhance training and regression metrics logging
- Added debug logging for data preparation, transformation, and prediction steps in the Training class to improve traceability.
- Updated compute_regression_metrics method to include metadata for better debugging and validation of index alignment between true and predicted values.
2026-04-13 15:18:42 -03:00
vitor-aignosi
388d4c95e4 chore: update configuration and imports for improved functionality
- Updated image tag in values.yaml from "1.0.0" to "1.0.2" for the latest version.
- Modified STORE_BASE_URL to include port 3000 for proper service access.
- Changed import from MinioRepository to MinioRepositorySync for synchronization support.
- Updated import from Postgres to PostgresSync to enhance experiment tracking capabilities.
- Renamed TRAIN_TASK_QUEUE from "train_model-single-queue" to "train_model-basic-queue" for clarity.
2026-04-13 14:57:06 -03:00
vitor-aignosi
4d761c15ac chore: remove sientia models module and relocate date format validation
- Deleted the sientia/models.py file, which contained the LinearRegressionModel and related functionality.
- Moved the frontend date format validation logic to train_model_params.py, ensuring a single source of truth for date formats.
2026-04-13 10:40:21 -03:00
vitor-aignosi
8765c46b5b chore: add jsonschema dependency to requirements.txt
- Included jsonschema version 4.26.0 to the requirements.txt file for enhanced validation capabilities.
2026-04-13 10:26:58 -03:00
vitor-aignosi
7ddae14b7d chore: update sientia-model-library version in requirements files
- Upgraded sientia-model-library from version 0.7.4 to 0.7.5 in both requirements-local.txt and requirements.txt.
- Modified quality-gate.yml to include sientia-model-library in the repositories list for improved CI/CD integration.
2026-04-13 09:05:55 -03:00
vitor-aignosi
bbf500e951 chore: remove git-requirements-mapping.txt as it is no longer needed 2026-04-13 08:47:37 -03:00
vitor-aignosi
ab34d09f65 chore: clarify ipykernel version constraint in requirements-dev.txt
- Updated comment for ipykernel to specify that other versions cause problems with interactive mode.
2026-04-13 08:43:12 -03:00
vitor-aignosi
5b41ef59e4 chore: ensure newline at end of requirements.txt for consistency 2026-04-13 08:10:10 -03:00
vitor-aignosi
9ea3249daf refactor: update GitHub repository links in requirements mapping and requirements files
- Changed repository links in git-requirements-mapping.txt to a more concise format.
- Updated requirements.txt to reference the sientia-dataops-library and sientia-model-library using simplified names instead of full GitHub URLs.
2026-04-13 08:04:56 -03:00
vitor-aignosi
62f5a88c5f chore: update dependency versions in requirements.txt
- Upgraded temporalio to version 1.23.0 for improved functionality.
- Updated sqlalchemy to version 2.0.49, and boto3 and botocore to version 1.42.70 for better compatibility and features.
2026-04-09 17:15:47 -03:00
vitor-aignosi
e2190081ff feat: update dependencies and refactor image configuration
- Replaced local paths in requirements.txt with specific GitHub repository links for sientia-dataops-library and sientia-model-library.
- Refactored image configuration in values.yaml to move it under the global section, ensuring consistent deployment settings across runtimes.
2026-04-09 17:12:28 -03:00
vitor-aignosi
c0bef2d688 feat: update values.yaml and refactor cleanup paths
- Changed project name in values.yaml from "sientia-dataops-model-manager" to "sientia-model-manager".
- Added new environment variables for GitHub repository and branch configuration.
- Refactored cleanup paths to use a centralized REPORTS_TEMP_DIR constant for consistency.
- Updated runtime configurations and adjusted volume mounts for better resource management.
- Enabled SSH access for the model manager and disabled Grafana dashboard creation.
- Updated tests to reflect changes in directory paths and environment variable usage.
2026-04-09 16:37:36 -03:00
vitor-aignosi
526edcb50e feat: update training workflow and repository management
- Replaced synchronous MinIO repository calls with asynchronous counterparts in the Training class for improved performance.
- Enhanced logging throughout the training process to provide better insights into model metadata loading, parameter validation, and training execution.
- Updated the train_test_split function to enforce DataFrame input type, ensuring consistency in data handling.
- Removed the deprecated model_repository.py file to streamline the codebase.
- Adjusted cleanup schedule logic to improve error handling and logging during schedule reconciliation.
- Updated tests to reflect changes in the training workflow and repository interactions.
2026-04-09 12:09:52 -03:00
vitor-aignosi
0ae03b246f feat: update environment configuration and remove deprecated model serving
- Modified `.env.example` to set local defaults for PostgreSQL, MLflow, and MinIO configurations.
- Added MongoDB configuration parameters to the environment setup.
- Updated `README.md` to reflect changes in workflow input parameters and task queue naming conventions.
- Removed the `ModelServing` class to streamline the codebase, as it was deemed unnecessary.
- Adjusted `connectors_config.py` to align with new environment variable names and improve clarity.
- Updated tests to reflect changes in configuration handling and removed tests related to the deleted `ModelServing` class.
2026-04-07 16:58:50 -03:00
vitor-aignosi
c5cd382350 feat: enhance configuration and scheduling for cleanup processes
- Updated `.env.example` to include new environment variables for MinIO and PyPI configuration.
- Refactored `create_cleanup_schedule` to utilize runtime-specific task queues and improve schedule reconciliation logic.
- Enhanced `Activities` class to require a default bucket in MinIO configuration.
- Adjusted `requirements.txt` to specify version for `evidently`.
- Updated tests to reflect changes in schedule creation and configuration handling.
2026-04-07 12:57:29 -03:00
vitor-aignosi
09ee92f100 feat: enhance configuration and error handling in project setup
- Added new ignore rule for Ruff to allow temporary paths in tests.
- Introduced MyPy overrides for specific modules to ignore errors.
- Refactored `Cleanup` and `ExperimentTracking` classes to remove async keywords from methods, improving consistency in method signatures.
- Updated `Training` class methods to handle synchronous operations, enhancing performance and clarity.
- Adjusted `requirements.txt` to remove unnecessary Git dependency, streamlining project setup.
2026-04-07 10:25:17 -03:00
vitor-aignosi
6b1df7c3a7 feat: enhance training and experiment tracking functionality
- Updated `Activities` class to improve garbage collection handling.
- Enhanced error messaging in `ExperimentTracking` for better clarity on update failures.
- Refactored `Training` class to streamline exception handling and improve type hints.
- Introduced new methods in `TrainModelParams` for better handling of experiment run IDs and model metadata.
- Added functionality to extract model equations in `DataManagerRepository` for linear regression models.
2026-04-06 15:05:57 -03:00
vitor-aignosi
1352d1ac8f feat: enhance training model functionality and reporting
- Added `evidently` to requirements for improved model evaluation.
- Introduced `TrainModelResult` class with a `to_dict` method for better result handling.
- Updated `train_model` method to return a comprehensive training result, including run details.
- Enhanced `cleanup_run_directory` method in `DataManagerRepository` for improved resource management.
- Adjusted type hints in `TrainModel` for clarity and consistency.
2026-04-06 11:40:08 -03:00
vitor-aignosi
a8b926649a feat: enhance training reporting and directory management
- Added functionality to store run name and ID in the training results.
- Implemented report generation in the `DataManagerRepository`, including methods to create a run directory and generate comprehensive reports.
- Updated `TrainModelResult` to include `run_id` and `run_dir` attributes for better tracking of training sessions.
2026-04-06 10:38:24 -03:00
vitor-aignosi
18412958d2 refactor: remove MinIO repository references and streamline cleanup process
- Eliminated MinIO repository dependencies from the `Activities`, `Cleanup`, and `Training` classes.
- Updated the `cleanup_resources` method to focus on removing local temporary directories instead of handling MinIO file deletions.
- Adjusted the `TrainModel` class to pass the run directory for cleanup, enhancing resource management in the training workflow.
2026-04-06 09:30:38 -03:00
vitor-aignosi
d2bb588739 Merge branch 'main' into release/SIENTIAPDE-1645 2026-04-06 08:34:15 -03:00
vitor-aignosi
bf2b6b3888 feat: improve error handling and resource cleanup in training workflow
- Updated the `Training` class to raise `ModelTrainingError` on training failures for better error management.
- Enhanced the `run` method in `TrainModel` to return training results and ensure proper resource cleanup, including validation files.
- Refactored exception handling to prevent silent failures during resource cleanup and experiment run updates.
- Adjusted type hints for improved clarity and consistency in method signatures.
2026-04-06 08:26:42 -03:00
github-actions[bot]
dfe05a063a chore: bump version to 1.2.0 [skip ci] 2026-03-30 18:27:46 +00:00
Bruno Domingues
f43fe6a399 Merge pull request #26 from Aignosi/feature/SIENTIAPDE-1717
SIENTIAPDE-1717: Remove MinIO Cleanup Functionality and Streamline Local Temporary File Management
2026-03-30 15:27:20 -03:00
Bruno Domingues
14f91b40aa SIENTIAPDE-1717: Updated documentation to reflect the removal of MinIO data download and cleanup operations from Model Manager workflows. 2026-03-30 14:54:44 -03:00
Bruno Domingues
cce350bfb1 SIENTIAPDE-1717: Remove FILE_DELETE_ERROR from ExperimentStatus enum and related tests. 2026-03-30 14:33:19 -03:00
Bruno Domingues
f5eb8d1e5b SIENTIAPDE-1717: Remove list_bucket_objects method from StorageRepository and its associated test. 2026-03-30 14:18:14 -03:00
Bruno Domingues
7a0961f29d SIENTIAPDE-1717: Remove MinIO cleanup functionality and associated components. This change streamlines the cleanup workflow to focus solely on local temporary directories, removes the ModelTrainingError exception, and updates related configurations, documentation, and tests. 2026-03-30 14:14:10 -03:00
vitor-aignosi
342a02d6f7 feat: enhance training workflow with model metadata loading and refactor data handling
- Introduced a new activity to load model metadata from the model store.
- Refactored training logic to utilize new model metadata and improved parameter handling.
- Updated the `TrainModelParams` class to include additional fields for model configuration.
- Replaced deprecated utility functions with a custom train-test split implementation.
- Removed unused utility functions and cleaned up the data manager repository.
- Adjusted experiment tracking to include model-specific metadata in notifications.
2026-03-24 14:39:28 -03:00
github-actions[bot]
63a94dae0a chore: bump version to 1.1.3 [skip ci] 2026-03-12 17:33:24 +00:00
Bruno Domingues
fb54ee4e9a Merge pull request #25 from Aignosi/fix/SIENTIAPDE-1672
SIENTIAPDE-1672: Streamline Deploy Workflow: Remove Versioning, Pass Branch, Enable VPN
2026-03-12 14:33:02 -03:00
Bruno Domingues
1d096a6dc0 SIENTIAPDE-1672: Streamline deploy workflow by removing dedicated versioning job, passing branch name to reusable deploy, and enabling VPN. 2026-03-12 13:17:07 -03:00
vitor-aignosi
cf5111e520 feat: integrate PluginStore and MinIO repository into model manager activities
- Added PluginStore integration for model management.
- Replaced StorageRepository with MinIORepository in Activities, Cleanup, and Training classes.
- Updated training logic to handle validation files and improved data management.
- Enhanced configuration for MinIO and PluginStore in connectors.
- Removed deprecated model repository and storage repository files.
- Updated environment variable handling for new configurations.
2026-03-11 17:35:05 -03:00
github-actions[bot]
9d71c0cf80 chore: bump version to 1.1.2 [skip ci] 2026-02-20 00:20:15 +00:00
Bruno Domingues
0a0adac86c Merge pull request #24 from Aignosi/fix/SIENTIAPDE-1579
Upgrade MLflow to 2.18.0 and constrain setuptools
2026-02-19 21:19:53 -03:00
Bruno Domingues
41c4490cad SIENTIAPDE-1579: Fix SientiaMlException propagation in ModelServing.search_runs_by_name to correctly raise the exception with its message, resolving a TypeError. Enhance training test script to support scenario-specific CSV files for different test cases. Update search_runs_by_name return type hint and apply minor code formatting. 2026-02-19 21:13:41 -03:00
Bruno Domingues
f952647cf6 SIENTIAPDE-1579: Update MLflow to 2.18.0 and add setuptools version constraint. 2026-02-19 17:56:08 -03:00
github-actions[bot]
b0629cb34a chore: bump version to 1.1.1 [skip ci] 2026-02-19 12:34:59 +00:00
Lucas Kou Kinoshita
8609245e77 Merge pull request #23 from Aignosi/fix/SIENTIAPDE-1579
Fix/SIENTIAPDE-1579: Corrigir bug durante o treinamento de certos arquivos no Soft Sensor
2026-02-19 09:34:34 -03:00
Kou Kinoshita
14f3fbadbe SIENTIAPDE-1579: Switched from replace logic to validation + mapping 2026-02-18 18:57:41 -03:00
Kou Kinoshita
e7e851dbb5 SIENTIAPDE-1579: Lint fix 2026-02-18 18:20:43 -03:00
Kou Kinoshita
5dab1ee576 SIENTIAPDE-1579?: removed duplicated method 2026-02-18 18:18:39 -03:00
Kou Kinoshita
a7922ac3ce SIENTIAPDE-1579: Lint fix 2026-02-18 17:50:58 -03:00