Commit Graph

19 Commits

Author SHA1 Message Date
PedroHMCosme
0c9c5f300b fix(worker): re-exec after install_runtime so imports match the runtime (SIENTIAPDE-2214)
The worker imports sklearn (metrics, evidently) before
PluginStore.install_runtime replaces it on disk with the runtime's pinned
version, the same order that breaks every sklearn prediction in laborious.
After installing the runtime, the worker now replaces its process once
(SIENTIA_RUNTIME_READY=1 skips the second install), so every module is
loaded from the runtime's packages.
2026-10-20 15:40:38 -03:00
c0bef2d688 feat: update values.yaml and refactor cleanup paths
- Changed project name in values.yaml from "sientia-dataops-model-manager" to "sientia-model-manager".
- Added new environment variables for GitHub repository and branch configuration.
- Refactored cleanup paths to use a centralized REPORTS_TEMP_DIR constant for consistency.
- Updated runtime configurations and adjusted volume mounts for better resource management.
- Enabled SSH access for the model manager and disabled Grafana dashboard creation.
- Updated tests to reflect changes in directory paths and environment variable usage.
2026-04-09 16:37:36 -03:00
c5cd382350 feat: enhance configuration and scheduling for cleanup processes
- Updated `.env.example` to include new environment variables for MinIO and PyPI configuration.
- Refactored `create_cleanup_schedule` to utilize runtime-specific task queues and improve schedule reconciliation logic.
- Enhanced `Activities` class to require a default bucket in MinIO configuration.
- Adjusted `requirements.txt` to specify version for `evidently`.
- Updated tests to reflect changes in schedule creation and configuration handling.
2026-04-07 12:57:29 -03:00
09ee92f100 feat: enhance configuration and error handling in project setup
- Added new ignore rule for Ruff to allow temporary paths in tests.
- Introduced MyPy overrides for specific modules to ignore errors.
- Refactored `Cleanup` and `ExperimentTracking` classes to remove async keywords from methods, improving consistency in method signatures.
- Updated `Training` class methods to handle synchronous operations, enhancing performance and clarity.
- Adjusted `requirements.txt` to remove unnecessary Git dependency, streamlining project setup.
2026-04-07 10:25:17 -03:00
6b1df7c3a7 feat: enhance training and experiment tracking functionality
- Updated `Activities` class to improve garbage collection handling.
- Enhanced error messaging in `ExperimentTracking` for better clarity on update failures.
- Refactored `Training` class to streamline exception handling and improve type hints.
- Introduced new methods in `TrainModelParams` for better handling of experiment run IDs and model metadata.
- Added functionality to extract model equations in `DataManagerRepository` for linear regression models.
2026-04-06 15:05:57 -03:00
cf5111e520 feat: integrate PluginStore and MinIO repository into model manager activities
- Added PluginStore integration for model management.
- Replaced StorageRepository with MinIORepository in Activities, Cleanup, and Training classes.
- Updated training logic to handle validation files and improved data management.
- Enhanced configuration for MinIO and PluginStore in connectors.
- Removed deprecated model repository and storage repository files.
- Updated environment variable handling for new configurations.
2026-03-11 17:35:05 -03:00
11aaa0780f SIENTIAPDE-1350: Improve Docker image build context and fix queue names
This commit refactors the .dockerignore file to use a whitelist approach, significantly reducing the Docker image size by excluding unnecessary files. It also fixes the queue names in values.yaml and adds TEMPORAL_USE_TLS to the test environment.
2025-11-26 17:52:44 -03:00
c281969381 SIENTIAPDE-1350: Add test to ensure worker starts despite cleanup schedule creation failure. 2025-11-26 11:30:22 -03:00
bc917e97d8 SIENTIAPDE-1350: Add cleanup schedule tests and configure worker with task queues from env vars 2025-11-25 19:21:55 -03:00
b028dc1b7d SIENTIAPDE-1350: Implement scheduled cleanup workflow using Temporal schedules. Adds schedule creation to worker startup and configures environment variables. 2025-11-25 18:12:53 -03:00
f3c88885ea SIENTIAPDE-1350: Configure cleanup test script and improve test isolation. This commit configures the cleanup test script to load environment variables from a .env file and use them for Temporal connection. It also adds a fixture to clean up temporary directories created by tests, ensuring better test isolation and preventing potential conflicts. Additionally, it adds the 'uri' property to the MongoDB config. 2025-11-25 00:44:55 -03:00
2ee66552d2 SIENTIAPDE-1350: Configure separate task queues for train and cleanup workers, update sientia-dataops-library to 1.6.1 and refactor prometheus server startup to use logger. 2025-11-24 11:21:43 -03:00
53f49d7a97 SIENTIAPDE-1350: Implement file cleanup workflow and activities, including MinIO and local directory cleanup, configuration, and metrics. 2025-11-19 18:38:00 -03:00
Kou-Kinoshita
a5c0cb6e47 SIENTIAPDE-1241: Fixed formatting and linted code 2025-10-30 10:38:29 -03:00
Kou-Kinoshita
7e9d6683e4 SIENTIAPDE-1241: Added worker test 2025-10-30 10:21:39 -03:00
Kou-Kinoshita
c699a47261 SIENTIAPDE-1241: worker tests 2025-10-30 09:52:34 -03:00
5789a13023 SIENTIAPDE-1241: refactor train_model workflow due to I/O errors. 2025-10-22 15:37:56 -03:00
7abd951806 SIENTIAPDE-1255: Refactor worker to support only the train_model-queue and remove prediction workflows. 2025-10-16 18:15:52 -03:00
2628dc92d2 SIENTIAPDE-1250: Refactor: Remove 'laborious' directory from test structure and update README.md accordingly. 2025-10-07 10:02:38 -03:00