fix(worker): set app_up only after the runtime version guard (SIENTIAPDE-2214)

The probes only check app_up, which was set to 1 when the metrics server started,
before install_runtime. A pod that would fail the guard could turn Ready first, so a
rollout would remove the old pod. app_up now goes to 1 right after the guard.

Also: copy sys.modules keys before iterating, raise the sientia_model floor to the
runtime's >=0.14.0, pin the same runtime versions in requirements-local.txt, test
that the snapshot is taken before the install and the real importlib.metadata path,
and document the boot failure in README and CLAUDE.md.
This commit is contained in:
PedroHMCosme
2026-10-22 12:17:12 -03:00
parent 0f5e4a5cef
commit dfbdd4b605
6 changed files with 74 additions and 15 deletions

View File

@@ -149,6 +149,7 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
- Prometheus metrics server initialization
- Notification handler setup and configuration
- **Runtime Installation**: Automatically installs the required model runtime from the Plugin Store
- **Runtime version guard**: boot fails if the install changes the version of a package the worker already imported, so the `requirements.txt` pins must match `model-plugin-factory` `runtime/basic.yaml` and `runtime/xgboost.yaml`; `app_up` goes to 1 only after this check
- **Key Features**:
- Automatic scaling with `PollerBehaviorAutoscaling`
- Health check endpoints for Kubernetes liveness/readiness probes
@@ -1249,6 +1250,10 @@ sientia-dataops-model-manager/
- Check training parameter validation errors
- Verify input data format and required fields
5. **Worker exits at boot with `Runtime ... changed packages already imported by the worker`**
- A runtime pin in `model-plugin-factory` `runtime/*.yaml` no longer matches `requirements.txt`; the message lists each package as `old -> new`
- Set the runtime version in `requirements.txt` and `requirements-local.txt` here and in `sientia-dataops-laborious`, then redeploy
### Debug Mode
Enable debug logging by setting the log level: