46 Commits

Author SHA1 Message Date
34eaa01ca7 SIENTIAPDE-2226: Stop installing runtime in worker; unpin requirements
- The sientia-module entrypoint now merges requirements.txt with the
  Plugin Store runtime and installs both before boot (SIENTIAPDE-2223),
  so the worker no longer calls install_runtime or runs the loaded
  package version guard.
- Leave runtime libs unpinned in requirements files so the runtime
  version wins; cap sqlalchemy<2.1 and evidently<0.7 for compatibility.
- Update README, CLAUDE.md and worker tests to match.
2026-10-27 10:43:41 -03:00
PedroHMCosme
dfbdd4b605 fix(worker): set app_up only after the runtime version guard (SIENTIAPDE-2214)
The probes only check app_up, which was set to 1 when the metrics server started,
before install_runtime. A pod that would fail the guard could turn Ready first, so a
rollout would remove the old pod. app_up now goes to 1 right after the guard.

Also: copy sys.modules keys before iterating, raise the sientia_model floor to the
runtime's >=0.14.0, pin the same runtime versions in requirements-local.txt, test
that the snapshot is taken before the install and the real importlib.metadata path,
and document the boot failure in README and CLAUDE.md.
2026-10-22 12:17:12 -03:00
PedroHMCosme
0f5e4a5cef fix(worker): pin the runtime versions and refuse to boot when install_runtime swaps an imported package (SIENTIAPDE-2214)
Replaces the re-exec. requirements.txt now pins the union of the basic and xgboost
runtime libraries, so install_runtime has nothing to change in the packages the worker
already imported. If a runtime pin changes and requirements.txt is not updated, the
worker logs which packages changed and exits before starting the workers.

test_main_entrypoint assigned worker.main = AsyncMock() without restoring it, so every
later test of main() ran the mock; it now uses patch.object.
2026-10-22 11:34:04 -03:00
PedroHMCosme
7903dda730 Merge branch 'main' into fix/SIENTIAPDE-2214 2026-10-21 09:28:48 -03:00
ced847bbe7 chore(ci): switch deploy.yml to manual workflow_dispatch with required chart_ref
Removes auto-deploy-on-PR-merge trigger and the old PR-merge gate. Deploy now
triggers only via workflow_dispatch, with chart_ref required (no default) as
the literal sientia-helm-chart branch bumped and deployed — hard error if it
doesn't exist, no silent fallback to development/main.

- .github/workflows/deploy.yml: drop pull_request trigger, add chart_ref
  input, branch_name now github.ref_name, drop pull-requests:read permission
- Delete .github/workflows/require-chart-ref-label.yml
- README.md: update Deploy section to describe manual trigger

[skip ci]
2026-10-20 17:03:33 -03:00
PedroHMCosme
262ba8d658 docs(worker): document SIENTIA_RUNTIME_READY, log the skipped install, test install failure (SIENTIAPDE-2214)
From the code-review-python pass on #34:
- README, CLAUDE.md and the module docstring: the worker re-executes
  itself after install_runtime, and SIENTIA_RUNTIME_READY is internal;
  setting it in .env/the chart would skip the install silently.
- "Installing runtime" is now logged only on the pass that installs; the
  re-executed pass logs that the install was skipped.
- The ponytail comment notes that the install success metrics are lost
  to the exec.
- New test: a failed install propagates without setting the flag or
  calling os.execv (locks the install -> flag -> exec order).
2026-10-20 16:30:23 -03:00
ebc7e5ce6f SIENTIAPDE-1645: Overhaul deployment, add Claude AI guidance, and enhance CI/CD.
Migrated to a shared sientia-module image and umbrella Helm chart, removing the standalone Dockerfile, .dockerignore, and values.yaml. Introduced CLAUDE.md for AI-assisted development. Updated quality-gate.yml with push triggers, concurrency, and refined permissions. Bumped project version to 2.0.0 to reflect these architectural changes.
2026-08-05 10:38:38 -03:00
159607fc72 SIENTIAPDE-1945: Adopt shared Docker image and Kubernetes deployment strategy.
Removed the local Dockerfile, .dockerignore, and values.yaml, as image building and Kubernetes deployment are now handled by external sientia-module components. Updated README accordingly.
2026-07-08 20:22:32 -03:00
ba9eb3d7c7 feat: require date_column in training parameters and update documentation
- Made `date_column` a required field in `TrainModelParams`, ensuring it must be present in the input data.
- Updated related documentation in `input-sample.md`, `README.md`, and various test scenarios to reflect the change in requirement.
- Adjusted the handling of `date_format` to default to `yyyy-MM-dd HH:mm:ss` if omitted, enhancing usability.
- Refined test scenarios to include new examples and ensure compliance with the updated parameter structure.

These changes improve the robustness of the model training workflow and clarify the expectations for input data.
2026-05-05 08:35:12 -03:00
50d0ea6f32 doc update 2026-04-17 12:12:58 -03:00
0ae03b246f feat: update environment configuration and remove deprecated model serving
- Modified `.env.example` to set local defaults for PostgreSQL, MLflow, and MinIO configurations.
- Added MongoDB configuration parameters to the environment setup.
- Updated `README.md` to reflect changes in workflow input parameters and task queue naming conventions.
- Removed the `ModelServing` class to streamline the codebase, as it was deemed unnecessary.
- Adjusted `connectors_config.py` to align with new environment variable names and improve clarity.
- Updated tests to reflect changes in configuration handling and removed tests related to the deleted `ModelServing` class.
2026-04-07 16:58:50 -03:00
14f91b40aa SIENTIAPDE-1717: Updated documentation to reflect the removal of MinIO data download and cleanup operations from Model Manager workflows. 2026-03-30 14:54:44 -03:00
7a0961f29d SIENTIAPDE-1717: Remove MinIO cleanup functionality and associated components. This change streamlines the cleanup workflow to focus solely on local temporary directories, removes the ModelTrainingError exception, and updates related configurations, documentation, and tests. 2026-03-30 14:14:10 -03:00
06571011f2 SIENTIAPDE-1430: Introduce static_threshold parameter for static window removal.
This parameter allows customizing the threshold (1-1000) used when rem_static_win is enabled, defaulting to 1 if null.
Updates include parameter definition, business rule validation, repository logic for passing the threshold, documentation in README.md and PIPELINE_PARAMS_CHANGELOG.md, and new unit and integration tests.
2025-12-19 15:44:18 -03:00
9a77257920 SIENTIAPDE-1430: Update README to document new features, expanded validation, and enhanced testing.
*   Documents support for Polynomial Regression, per-variable lag configuration, date range filtering, and NaN treatment options.
*   Includes details on RCE drift metrics and comprehensive business validation rules for training parameters.
*   Describes new JSON-based integration test scenarios with batch execution and updated 100% code coverage.
*   Refines architecture overview and example workflow parameters.
2025-12-19 11:54:08 -03:00
06fd08dc70 SIENTIAPDE-1430: Introduce comprehensive integration testing with JSON-based scenarios and detailed README documentation. Enhance training workflow to support advanced model configurations, including polynomial regression with mandatory scaler validation. Ensure robust prediction handling by calculating training predictions (y_train_pred) before denormalization and automatically configuring datetime indices for time-series operations. 2025-12-18 17:05:10 -03:00
32aa66e5d8 SIENTIAPDE-1352: Update README with improved instructions, project structure details, and environment variable descriptions. 2025-12-04 19:31:35 -03:00
5fb8aead22 SIENTIAPDE-1352: Update deploy workflow to use reusable workflow and add permissions, update README with CI/CD details. 2025-12-04 12:54:03 -03:00
cdaafd12cc SIENTIAPDE-1352: Add deploy workflow and update README with local repository execution instructions. 2025-12-04 12:09:50 -03:00
46df241f7d SIENTIAPDE-1352: Add documentation for local GitHub Actions testing with act.
This commit adds a new section to the README.md file detailing how to use the 'act' tool for local testing of GitHub Actions workflows. It includes installation instructions, configuration details, usage examples, and troubleshooting tips.
2025-12-02 17:28:04 -03:00
486e1756c9 SIENTIAPDE-1350: Implement automated file cleanup workflow and update documentation. Added a new workflow for automated cleanup of stale files from MinIO and local filesystem, including scheduled execution and configurable retention. Updated the README with detailed information about the new workflow, its configuration, and related metrics. (118 additions, 5 deletions) 2025-11-27 10:22:38 -03:00
7ff54a1112 SIENTIAPDE-1309: Improve Dockerignore, Gitignore, Dockerfile, README, model repository, tests, todo list and values.yaml files. 2025-11-07 15:08:50 -03:00
c7f44a2423 SIENTIAPDE-1309: Update README with Helm instructions and refactor experiment status messages. Also, update values.yaml with new image and configurations. 2025-11-06 15:34:03 -03:00
a625ef6c19 SIENTIAPDE-1309: Add Helm chart documentation to README and update image tag in values.yaml 2025-11-05 21:26:58 -03:00
d9d3df5723 SIENTIAPDE-1308: Initial Docker setup, improved .dockerignore, updated README, and minor fixes. 2025-11-04 14:07:37 -03:00
9f56a128e6 SIENTIAPDE-1308: Update documentation, remove install_dependencies.sh script from README, correct Training class initialization, and update sientia-dataops-library version in requirements.txt. 2025-11-03 17:35:33 -03:00
Kou-Kinoshita
63fa264ec4 SIENTIAPDE-1241: Updated readme (removed port foward script)) 2025-10-29 16:27:56 -03:00
9e31ab679b SIENTIAPDE-1255: Implement MLFlow artifact management and model persistence
This commit introduces a new model_repository.py to handle MLFlow artifact generation and model persistence. It also updates the README to reflect this change and modifies training_repository.py to separate training and MLFlow operations.
2025-10-17 09:11:54 -03:00
a66b996cc1 SIENTIAPDE-1255: Refactor MLFlow activities for training operations and update metrics
This commit refactors the MLFlow activities to focus on model training rather than prediction operations. It removes prediction-related activities and metrics, and updates the MLFlow activity descriptions to reflect the change in focus. The README is also updated to reflect these changes.
2025-10-17 00:51:29 -03:00
85a8ba1e68 SIENTIAPDE-1255: Refactor data quality gates to training focused metrics and repositories. This commit removes the data quality gates and filters, focusing on training-specific metrics and data repositories. It also updates the README to reflect these changes, including new training metrics and a streamlined data services section. 2025-10-17 00:39:06 -03:00
e191e7849f SIENTIAPDE-1255: Refactor Model Manager to focus on ML model training pipeline and simplify architecture. This includes removing prediction workflows, updating core functionality, and improving parameter validation and resource management.
README.md | 411 deletions(-) 117 insertions(+)
1 file changed, 117 insertions(+), 411 deletions(-)
2025-10-16 18:26:56 -03:00
8f1cc21bb1 SIENTIAPDE-1255: Add port forwarding setup script and documentation to README.md 2025-10-16 18:00:28 -03:00
4ff796a56f SIENTIAPDE-1253: Update README.md to include documentation for the Train Model workflow and renumber the Minimal Retrain workflow. (+101, -1) 2025-10-15 16:35:13 -03:00
780184769d SIENTIAPDE-1253: Expose workflow activity timeouts as environment variables and document them. 2025-10-15 16:03:01 -03:00
bee6036205 SIENTIAPDE-1251: Implement ML model training activity and repository
This commit introduces the 'Training' activity and 'TrainingRepository' for handling ML model training operations within the Model Manager system.

- Added model_manager/activities/training.py for the Training activity, which extends BaseActivity and integrates with Temporal workflows.
- Added model_manager/utils/repository/training_repository.py for the TrainingRepository, which encapsulates the core training logic.
- Updated model_manager/activities/activities.py to include the Training activity in the main activities orchestrator.
- Updated README.md to document the new 'Training' component.
- Added unit tests for the new activity and repository.
2025-10-09 15:07:27 -03:00
7e1eace77f SIENTIAPDE-1250: Implement experiment tracking activity with status updates, error handling, and model registration. This includes a unified update method, error message truncation, and integration into the main activities orchestrator. (236+, 2-) 2025-10-07 11:27:23 -03:00
2628dc92d2 SIENTIAPDE-1250: Refactor: Remove 'laborious' directory from test structure and update README.md accordingly. 2025-10-07 10:02:38 -03:00
ec5b9df125 SIENTIAPDE-1248: Refactor README.md to include detailed documentation on features, architecture, workflows, installation, code quality, testing, monitoring, configuration, development, troubleshooting, performance tuning, contributing, license, and support. 2025-10-04 14:45:42 -03:00
9a1355fe9b SIENTIAPDE-1243: Add Temporal namespace setup guide to README and improve validation script. 2025-10-02 10:08:02 -03:00
8f4ac3284e SIENTIAPDE-1243: Remove Redis dependency and related configurations. 2025-10-01 18:16:21 -03:00
b102f79087 SIENTIAPDE-1243: Remove OPC server integration and add code quality tools.
This commit removes the OPC server integration from the Model Manager, including related activities, repositories, metrics, and configuration. It also adds code quality tools such as Ruff (linting/formatting), mypy (type checking), and Bandit (security analysis) along with a validation script and CI/CD integration for automated code validation. The README has been updated to reflect these changes.
2025-10-01 16:25:08 -03:00
bc4d98f78d SIENTIAPDE-1243: Rename project from 'laborious' to 'model-manager' across codebase and configuration. This includes updating project names in environment variables, Makefiles, README, metrics, and Helm chart values. 2025-10-01 14:41:24 -03:00
aba4a3f1a5 SIENTIAPDE-1243: Refactor: Rename 'laborious' package to 'model_manager'
This commit renames the 'laborious' package to 'model_manager' across the entire project. This includes renaming directories, modules, references in code, configuration files, and documentation to reflect the new package name. This change improves clarity and consistency within the project.
2025-10-01 14:29:24 -03:00
e318b91c63 SIENTIAPDE-1243: Update README with correct repository name and installation instructions. 2025-10-01 14:11:36 -03:00
93d0849c80 SIENTIAPDE-1243: Initial commit of the model manager project, adding core files and configurations.
This commit introduces the initial project structure, including:
- .env.example: Example environment configuration.
- .github/workflows/quality-gate.yml: CI workflow for quality checks.
- .gitignore: Specifies intentionally untracked files that Git should ignore.
- Makefile: Automation of tasks like docker builds.
- README.md: Project documentation.
- Source code for model management, activities, utils, worker and workflows.
- Test suite.
- Dockerfile for the simulator.
- sonar-project.properties: SonarQube configuration file.
- values.yaml: Helm chart values for deployment.
2025-09-30 14:55:38 -03:00
Bruno Domingues
e79b35a1fb Initial commit 2025-09-30 13:38:26 -03:00