Commit Graph

296 Commits

Author SHA1 Message Date
PedroHMCosme
29700169ad refactor(worker): iterate a sys.modules copy instead of list() (Sonar S7504) (SIENTIAPDE-2214) 2026-10-22 12:37:11 -03:00
PedroHMCosme
dfbdd4b605 fix(worker): set app_up only after the runtime version guard (SIENTIAPDE-2214)
The probes only check app_up, which was set to 1 when the metrics server started,
before install_runtime. A pod that would fail the guard could turn Ready first, so a
rollout would remove the old pod. app_up now goes to 1 right after the guard.

Also: copy sys.modules keys before iterating, raise the sientia_model floor to the
runtime's >=0.14.0, pin the same runtime versions in requirements-local.txt, test
that the snapshot is taken before the install and the real importlib.metadata path,
and document the boot failure in README and CLAUDE.md.
2026-10-22 12:17:12 -03:00
PedroHMCosme
0f5e4a5cef fix(worker): pin the runtime versions and refuse to boot when install_runtime swaps an imported package (SIENTIAPDE-2214)
Replaces the re-exec. requirements.txt now pins the union of the basic and xgboost
runtime libraries, so install_runtime has nothing to change in the packages the worker
already imported. If a runtime pin changes and requirements.txt is not updated, the
worker logs which packages changed and exits before starting the workers.

test_main_entrypoint assigned worker.main = AsyncMock() without restoring it, so every
later test of main() ran the mock; it now uses patch.object.
2026-10-22 11:34:04 -03:00
PedroHMCosme
993c7d7662 Merge remote-tracking branch 'origin/main' into fix/SIENTIAPDE-2214 2026-10-21 21:46:49 -03:00
dc9fd871bb fix(ci): restaura pull-requests:read no deploy.yml, exigido pelo release-tarball
reusable-deploy-dataops.yml encadeia release-tarball -> dataops-module-release.yml,
que pede pull-requests:read; GitHub valida a interseção de permissions em todo o
workflow_call encadeado, independente do if: que gate o job. Removido por engano
em 586a5b9, achando que só existia por causa do trigger pull_request antigo.

[skip ci]
2026-10-21 14:10:25 -03:00
242566f9b1 fix(ci): grant contents:write permission to version-bump workflow
Caller workflow had no permissions block, so GitHub's permission-
intersection rule capped the nested reusable-version-bump.yml call
to read-only, breaking dispatch ("requesting 'contents: write', but
is only allowed 'contents: read'").

[skip ci]
2026-10-21 11:44:53 -03:00
PedroHMCosme
7903dda730 Merge branch 'main' into fix/SIENTIAPDE-2214 2026-10-21 09:28:48 -03:00
d0b4644f63 chore(ci): add version-bump.yml, decoupled from deploy
New workflow bumps pyproject.toml's version on PR merge, calling the
shared reusable-version-bump.yml (SIENTIAPDE-2217 version-bump reform) —
zero build/image/chart/deploy side effects. deploy.yml (workflow_dispatch-
only) will read this committed value instead of calculating one.

[skip ci]
2026-10-21 00:18:51 -03:00
ced847bbe7 chore(ci): switch deploy.yml to manual workflow_dispatch with required chart_ref
Removes auto-deploy-on-PR-merge trigger and the old PR-merge gate. Deploy now
triggers only via workflow_dispatch, with chart_ref required (no default) as
the literal sientia-helm-chart branch bumped and deployed — hard error if it
doesn't exist, no silent fallback to development/main.

- .github/workflows/deploy.yml: drop pull_request trigger, add chart_ref
  input, branch_name now github.ref_name, drop pull-requests:read permission
- Delete .github/workflows/require-chart-ref-label.yml
- README.md: update Deploy section to describe manual trigger

[skip ci]
2026-10-20 17:03:33 -03:00
PedroHMCosme
262ba8d658 docs(worker): document SIENTIA_RUNTIME_READY, log the skipped install, test install failure (SIENTIAPDE-2214)
From the code-review-python pass on #34:
- README, CLAUDE.md and the module docstring: the worker re-executes
  itself after install_runtime, and SIENTIA_RUNTIME_READY is internal;
  setting it in .env/the chart would skip the install silently.
- "Installing runtime" is now logged only on the pass that installs; the
  re-executed pass logs that the install was skipped.
- The ponytail comment notes that the install success metrics are lost
  to the exec.
- New test: a failed install propagates without setting the flag or
  calling os.execv (locks the install -> flag -> exec order).
2026-10-20 16:30:23 -03:00
PedroHMCosme
0c9c5f300b fix(worker): re-exec after install_runtime so imports match the runtime (SIENTIAPDE-2214)
The worker imports sklearn (metrics, evidently) before
PluginStore.install_runtime replaces it on disk with the runtime's pinned
version, the same order that breaks every sklearn prediction in laborious.
After installing the runtime, the worker now replaces its process once
(SIENTIA_RUNTIME_READY=1 skips the second install), so every module is
loaded from the runtime's packages.
2026-10-20 15:40:38 -03:00
49c8942a87 chore(ci): remove feature/** do push trigger (evita run duplicado) [skip ci] 2026-10-19 15:38:47 -03:00
github-actions[bot]
3b90812c12 chore: bump version to 2.0.3 2026-10-16 15:10:26 +00:00
0b673b35a1 Merge pull request #33 from Aignosi/fix/SIENTIAPDE-2201
SIENTIAPDE-2201: Keep the timestamp column in train/test data CSVs
2026-10-16 12:10:01 -03:00
PedroHMCosme
37f16f34ff SIENTIAPDE-2201: fix: keep the timestamp column in train/test data CSVs
generate_report wrote train_data.csv and test_data.csv with index=False, which
dropped the UTC time index. laborious drift reads test_data.csv from MLflow as
its reference and needs a timestamp column; without it, every model trained here
drifted against the first 30% of its own target window. Both CSVs now write the
index as timestamp. If a column already has that name, the CSV is written as
before and a warning is logged, so training never fails over drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 10:09:06 -03:00
dd981b8027 fix(ci): Update env_list_path to array format in deploy workflow
This commit changes the env_list_path parameter in the deploy GitHub Actions workflow to an array format, aligning with expected input types for the action.
2026-09-11 00:41:42 -03:00
github-actions[bot]
5641734c3c chore: bump version to 2.0.2 2026-09-10 13:48:02 +00:00
Bruno Domingues
c51466f304 Merge pull request #32 from Aignosi/fix/SIENTIAPDE-2068
Configure deploy workflow for automated version updates
2026-09-10 10:47:41 -03:00
44a4ffe6d0 SIENTIAPDE-2068: Configure deploy workflow to update version in pyproject.toml. 2026-09-10 10:39:49 -03:00
Bruno Domingues
3f5f61dd07 Merge pull request #31 from Aignosi/fix/SIENTIAPDE-2068
CI/CD: Migrate Model Manager Deployment to Flow B
2026-09-09 22:11:09 -03:00
274192a5f3 SIENTIAPDE-2068: feat(cicd): migrate model-manager deploy to Flow B
- Replace legacy reusable-deploy.yml call (stale input shape, pointed at
  sientia-dataops-helm-repo) with reusable-deploy-dataops.yml — mirror to
  Gitea, dual-bump GITHUB_BRANCH (development + chart-ref label branch),
  helmfile deploy. First tarball release for this repo (create_tarball
  default true). workflow_dispatch has no inputs.
- Add require-chart-ref-label.yml.
2026-09-09 21:49:46 -03:00
c464681db5 Merge pull request #29 from Aignosi/fix/SIENTIAPDE-2072
SIENTIAPDE-2072: bump sientia_do pin to 1.12.2
2026-09-09 11:53:06 -03:00
20cb9827b0 Merge branch 'main' into fix/SIENTIAPDE-2072 2026-09-09 10:34:51 -03:00
github-actions[bot]
22d0c9c342 chore: bump version to 2.0.1 [skip ci] 2026-08-20 13:47:38 +00:00
a7d3b43db9 Merge pull request #30 from Aignosi/fix/model-library-version
Bump sientia-model-library to 0.14.0
2026-08-20 10:47:16 -03:00
5048ecbdd4 model-library-version: Update sientia-model-library from 0.8.3 to 0.14.0 in local requirements. 2026-08-20 10:40:02 -03:00
Eduardo Rios
ae8066cb26 SIENTIAPDE-2072: bump sientia_do pin to 1.12.2
Picks up the notification timestamp -> native datetime fix; also
aligns requirements.txt's floor with requirements-local.txt's pin
(was >=1.11.0/@1.12.0, out of sync with the other repos).
2026-08-17 16:42:24 -03:00
Bruno Domingues
85642bed25 Merge pull request #27 from Aignosi/release/SIENTIAPDE-1645
Comprehensive Updates: Training Workflow, Configuration, and Dependency Management
2026-08-05 13:17:43 -03:00
e3e1c712a8 SIENTIAPDE-1645: Refactor temporary directory cleanup activity by extracting processing and deletion logic into dedicated helper methods. Update train_test_split to use a local numpy.random generator for improved reproducibility. Remove an unnecessary else: pass block. 2026-08-05 12:10:54 -03:00
4133e200f7 SIENTIAPDE-1645: Remove validate.sh script, add jsonschema dependency, and improve test code readability 2026-08-05 11:23:22 -03:00
ebc7e5ce6f SIENTIAPDE-1645: Overhaul deployment, add Claude AI guidance, and enhance CI/CD.
Migrated to a shared sientia-module image and umbrella Helm chart, removing the standalone Dockerfile, .dockerignore, and values.yaml. Introduced CLAUDE.md for AI-assisted development. Updated quality-gate.yml with push triggers, concurrency, and refined permissions. Bumped project version to 2.0.0 to reflect these architectural changes.
2026-08-05 10:38:38 -03:00
github-actions[bot]
b6e59cf18e chore: bump version to 1.3.0 [skip ci] 2026-07-10 14:48:08 +00:00
Bruno Domingues
3457496c0a Merge pull request #28 from Aignosi/feature/SIENTIAPDE-1945
SIENTIAPDE-1945: Adopt Shared Docker/Kubernetes Deployment Strategy
2026-07-10 11:47:50 -03:00
159607fc72 SIENTIAPDE-1945: Adopt shared Docker image and Kubernetes deployment strategy.
Removed the local Dockerfile, .dockerignore, and values.yaml, as image building and Kubernetes deployment are now handled by external sientia-module components. Updated README accordingly.
2026-07-08 20:22:32 -03:00
1c06a8e584 ci(sonar): Update project key 2026-07-01 21:07:47 -03:00
2c63414770 SIENTIAPDE-1645: Expose detailed model training information via a new Prometheus gauge. This gauge, sientia_training_info, records metadata such as dataset sizes, feature count, and evaluation metrics (MSE, MAE, R2) along with the training run's timestamp. 2026-06-18 09:56:30 -03:00
bc65ebb955 SIENTIAPDE-1645: Use absolute time (time.time()) for training start timestamps instead of monotonic time. 2026-06-17 17:26:34 -03:00
214923b93e SIENTIAPDE-1645: Update pod_id label retrieval in training activities to use HOSTNAME environment variable with localhost fallback. 2026-06-17 16:12:51 -03:00
fe8a9384ba SIENTIAPDE-1645: Rename Prometheus 'sientia_app_up' metric to 'app_up' and update test assertion. 2026-06-17 12:26:25 -03:00
ea98032057 SIENTIAPDE-1645: Upgrade sqlalchemy to version 2.0.50 in requirements.txt 2026-06-17 11:20:23 -03:00
76f926a8ab SIENTIAPDE-1645: Add comprehensive model training observability metrics and Grafana dashboard. 2026-06-17 10:34:56 -03:00
445fe643fe chore: update .gitignore and requirements for development
- Added new entries to .gitignore to exclude temporary files and training input datasets, ensuring a cleaner repository.
- Included type stubs for psycopg2 in requirements-dev.txt to enhance type checking support for database interactions.
- Refactored the run_training_test.py script to implement a structured approach for loading and validating training input JSON files, improving the robustness of the training workflow.
2026-05-26 09:38:24 -03:00
64295d3855 refactor: update runtime configuration in values.yaml
- Changed the structure of the `runtimes` section in `values.yaml` to use a key-value format for better clarity and organization, replacing the previous list format.
- Updated the names of the runtimes to be keys, improving readability and maintainability of the configuration.
2026-05-22 16:25:20 -03:00
bcd4fab259 refactor: update input dataset and training parameters
- Renamed columns in `input_dataset.csv` from `feature_a`, `feature_b`, and `target` to `Counter`, `Rollout`, and `Square` for better clarity.
- Updated the `run_training_test.py` script to reflect the new column names in the workflow input, ensuring consistency in data processing.
- Added `date_column` parameter to the workflow input for improved data handling.
2026-05-11 12:49:04 -03:00
544c768ef4 chore: update sientia library versions in requirements files
- Updated `sientia-dataops-library` to version 1.12.0 and `sientia-model-library` to version 0.8.3 in `requirements-local.txt` for improved features and compatibility.
- Changed `sientia_do` and `sientia_model` in `requirements.txt` to use version specifiers for better flexibility in dependency management.
2026-05-06 17:15:16 -03:00
78167f7a44 fix: downgrade sientia_do version and update CSV saving logic
- Downgraded `sientia_do` version to 1.11.0 in `requirements.txt` to address compatibility issues.
- Updated `data_manager_repository.py` to save training and test data CSVs using float-cast versions of the data, ensuring consistency in the saved outputs.
- Enhanced test coverage in `test_data_manager_repository.py` to verify the presence of the target alias in the generated CSV files.
2026-05-05 16:26:47 -03:00
23912d854e chore: update dependencies and improve import statements
- Updated `sientia_do` version to 1.12.0 in `requirements.txt` for compatibility and feature enhancements.
- Refined import statements in `conftest.py` to improve type checking, ensuring better compatibility with type checkers.
2026-05-05 15:52:39 -03:00
ad125e22e1 chore: update .gitignore to include models directory
- Added 'models/' to .gitignore to prevent tracking of model files, ensuring a cleaner repository and avoiding unnecessary clutter in version control.
2026-05-05 11:00:14 -03:00
d1f9394879 feat: enhance E2E testing setup and model reporting
- Added a new fixture to manage runtime report artifacts in a writable temp directory during E2E tests, addressing permission issues in local CI/dev environments.
- Updated `conftest.py` to include a requirements.txt file in the model packaging path for training activities.
- Refactored existing fixtures to use `pytest.fixture` instead of `pytest_asyncio.fixture` for better compatibility.
- Enhanced the `Reports` class to include a target alias for report metrics, ensuring compatibility with Evidently's reporting requirements.
- Introduced new test scenarios to validate the handling of missing and whitespace-only `date_column` inputs in the training workflow.

These changes improve the robustness of the E2E testing framework and enhance the clarity of model reporting metrics.
2026-05-05 10:59:51 -03:00
ba9eb3d7c7 feat: require date_column in training parameters and update documentation
- Made `date_column` a required field in `TrainModelParams`, ensuring it must be present in the input data.
- Updated related documentation in `input-sample.md`, `README.md`, and various test scenarios to reflect the change in requirement.
- Adjusted the handling of `date_format` to default to `yyyy-MM-dd HH:mm:ss` if omitted, enhancing usability.
- Refined test scenarios to include new examples and ensure compliance with the updated parameter structure.

These changes improve the robustness of the model training workflow and clarify the expectations for input data.
2026-05-05 08:35:12 -03:00