Commit Graph

23 Commits

Author SHA1 Message Date
34eaa01ca7 SIENTIAPDE-2226: Stop installing runtime in worker; unpin requirements
- The sientia-module entrypoint now merges requirements.txt with the
  Plugin Store runtime and installs both before boot (SIENTIAPDE-2223),
  so the worker no longer calls install_runtime or runs the loaded
  package version guard.
- Leave runtime libs unpinned in requirements files so the runtime
  version wins; cap sqlalchemy<2.1 and evidently<0.7 for compatibility.
- Update README, CLAUDE.md and worker tests to match.
2026-10-27 10:43:41 -03:00
PedroHMCosme
dfbdd4b605 fix(worker): set app_up only after the runtime version guard (SIENTIAPDE-2214)
The probes only check app_up, which was set to 1 when the metrics server started,
before install_runtime. A pod that would fail the guard could turn Ready first, so a
rollout would remove the old pod. app_up now goes to 1 right after the guard.

Also: copy sys.modules keys before iterating, raise the sientia_model floor to the
runtime's >=0.14.0, pin the same runtime versions in requirements-local.txt, test
that the snapshot is taken before the install and the real importlib.metadata path,
and document the boot failure in README and CLAUDE.md.
2026-10-22 12:17:12 -03:00
PedroHMCosme
0f5e4a5cef fix(worker): pin the runtime versions and refuse to boot when install_runtime swaps an imported package (SIENTIAPDE-2214)
Replaces the re-exec. requirements.txt now pins the union of the basic and xgboost
runtime libraries, so install_runtime has nothing to change in the packages the worker
already imported. If a runtime pin changes and requirements.txt is not updated, the
worker logs which packages changed and exits before starting the workers.

test_main_entrypoint assigned worker.main = AsyncMock() without restoring it, so every
later test of main() ran the mock; it now uses patch.object.
2026-10-22 11:34:04 -03:00
PedroHMCosme
262ba8d658 docs(worker): document SIENTIA_RUNTIME_READY, log the skipped install, test install failure (SIENTIAPDE-2214)
From the code-review-python pass on #34:
- README, CLAUDE.md and the module docstring: the worker re-executes
  itself after install_runtime, and SIENTIA_RUNTIME_READY is internal;
  setting it in .env/the chart would skip the install silently.
- "Installing runtime" is now logged only on the pass that installs; the
  re-executed pass logs that the install was skipped.
- The ponytail comment notes that the install success metrics are lost
  to the exec.
- New test: a failed install propagates without setting the flag or
  calling os.execv (locks the install -> flag -> exec order).
2026-10-20 16:30:23 -03:00
PedroHMCosme
0c9c5f300b fix(worker): re-exec after install_runtime so imports match the runtime (SIENTIAPDE-2214)
The worker imports sklearn (metrics, evidently) before
PluginStore.install_runtime replaces it on disk with the runtime's pinned
version, the same order that breaks every sklearn prediction in laborious.
After installing the runtime, the worker now replaces its process once
(SIENTIA_RUNTIME_READY=1 skips the second install), so every module is
loaded from the runtime's packages.
2026-10-20 15:40:38 -03:00
c0bef2d688 feat: update values.yaml and refactor cleanup paths
- Changed project name in values.yaml from "sientia-dataops-model-manager" to "sientia-model-manager".
- Added new environment variables for GitHub repository and branch configuration.
- Refactored cleanup paths to use a centralized REPORTS_TEMP_DIR constant for consistency.
- Updated runtime configurations and adjusted volume mounts for better resource management.
- Enabled SSH access for the model manager and disabled Grafana dashboard creation.
- Updated tests to reflect changes in directory paths and environment variable usage.
2026-04-09 16:37:36 -03:00
c5cd382350 feat: enhance configuration and scheduling for cleanup processes
- Updated `.env.example` to include new environment variables for MinIO and PyPI configuration.
- Refactored `create_cleanup_schedule` to utilize runtime-specific task queues and improve schedule reconciliation logic.
- Enhanced `Activities` class to require a default bucket in MinIO configuration.
- Adjusted `requirements.txt` to specify version for `evidently`.
- Updated tests to reflect changes in schedule creation and configuration handling.
2026-04-07 12:57:29 -03:00
09ee92f100 feat: enhance configuration and error handling in project setup
- Added new ignore rule for Ruff to allow temporary paths in tests.
- Introduced MyPy overrides for specific modules to ignore errors.
- Refactored `Cleanup` and `ExperimentTracking` classes to remove async keywords from methods, improving consistency in method signatures.
- Updated `Training` class methods to handle synchronous operations, enhancing performance and clarity.
- Adjusted `requirements.txt` to remove unnecessary Git dependency, streamlining project setup.
2026-04-07 10:25:17 -03:00
6b1df7c3a7 feat: enhance training and experiment tracking functionality
- Updated `Activities` class to improve garbage collection handling.
- Enhanced error messaging in `ExperimentTracking` for better clarity on update failures.
- Refactored `Training` class to streamline exception handling and improve type hints.
- Introduced new methods in `TrainModelParams` for better handling of experiment run IDs and model metadata.
- Added functionality to extract model equations in `DataManagerRepository` for linear regression models.
2026-04-06 15:05:57 -03:00
cf5111e520 feat: integrate PluginStore and MinIO repository into model manager activities
- Added PluginStore integration for model management.
- Replaced StorageRepository with MinIORepository in Activities, Cleanup, and Training classes.
- Updated training logic to handle validation files and improved data management.
- Enhanced configuration for MinIO and PluginStore in connectors.
- Removed deprecated model repository and storage repository files.
- Updated environment variable handling for new configurations.
2026-03-11 17:35:05 -03:00
11aaa0780f SIENTIAPDE-1350: Improve Docker image build context and fix queue names
This commit refactors the .dockerignore file to use a whitelist approach, significantly reducing the Docker image size by excluding unnecessary files. It also fixes the queue names in values.yaml and adds TEMPORAL_USE_TLS to the test environment.
2025-11-26 17:52:44 -03:00
c281969381 SIENTIAPDE-1350: Add test to ensure worker starts despite cleanup schedule creation failure. 2025-11-26 11:30:22 -03:00
bc917e97d8 SIENTIAPDE-1350: Add cleanup schedule tests and configure worker with task queues from env vars 2025-11-25 19:21:55 -03:00
b028dc1b7d SIENTIAPDE-1350: Implement scheduled cleanup workflow using Temporal schedules. Adds schedule creation to worker startup and configures environment variables. 2025-11-25 18:12:53 -03:00
f3c88885ea SIENTIAPDE-1350: Configure cleanup test script and improve test isolation. This commit configures the cleanup test script to load environment variables from a .env file and use them for Temporal connection. It also adds a fixture to clean up temporary directories created by tests, ensuring better test isolation and preventing potential conflicts. Additionally, it adds the 'uri' property to the MongoDB config. 2025-11-25 00:44:55 -03:00
2ee66552d2 SIENTIAPDE-1350: Configure separate task queues for train and cleanup workers, update sientia-dataops-library to 1.6.1 and refactor prometheus server startup to use logger. 2025-11-24 11:21:43 -03:00
53f49d7a97 SIENTIAPDE-1350: Implement file cleanup workflow and activities, including MinIO and local directory cleanup, configuration, and metrics. 2025-11-19 18:38:00 -03:00
Kou-Kinoshita
a5c0cb6e47 SIENTIAPDE-1241: Fixed formatting and linted code 2025-10-30 10:38:29 -03:00
Kou-Kinoshita
7e9d6683e4 SIENTIAPDE-1241: Added worker test 2025-10-30 10:21:39 -03:00
Kou-Kinoshita
c699a47261 SIENTIAPDE-1241: worker tests 2025-10-30 09:52:34 -03:00
5789a13023 SIENTIAPDE-1241: refactor train_model workflow due to I/O errors. 2025-10-22 15:37:56 -03:00
7abd951806 SIENTIAPDE-1255: Refactor worker to support only the train_model-queue and remove prediction workflows. 2025-10-16 18:15:52 -03:00
2628dc92d2 SIENTIAPDE-1250: Refactor: Remove 'laborious' directory from test structure and update README.md accordingly. 2025-10-07 10:02:38 -03:00