fix(worker): set app_up only after the runtime version guard (SIENTIAPDE-2214)
The probes only check app_up, which was set to 1 when the metrics server started, before install_runtime. A pod that would fail the guard could turn Ready first, so a rollout would remove the old pod. app_up now goes to 1 right after the guard. Also: copy sys.modules keys before iterating, raise the sientia_model floor to the runtime's >=0.14.0, pin the same runtime versions in requirements-local.txt, test that the snapshot is taken before the install and the real importlib.metadata path, and document the boot failure in README and CLAUDE.md.
This commit is contained in:
@@ -149,6 +149,7 @@ The Model Manager system uses a Temporal-based workflow architecture with clear
|
||||
- Prometheus metrics server initialization
|
||||
- Notification handler setup and configuration
|
||||
- **Runtime Installation**: Automatically installs the required model runtime from the Plugin Store
|
||||
- **Runtime version guard**: boot fails if the install changes the version of a package the worker already imported, so the `requirements.txt` pins must match `model-plugin-factory` `runtime/basic.yaml` and `runtime/xgboost.yaml`; `app_up` goes to 1 only after this check
|
||||
- **Key Features**:
|
||||
- Automatic scaling with `PollerBehaviorAutoscaling`
|
||||
- Health check endpoints for Kubernetes liveness/readiness probes
|
||||
@@ -1249,6 +1250,10 @@ sientia-dataops-model-manager/
|
||||
- Check training parameter validation errors
|
||||
- Verify input data format and required fields
|
||||
|
||||
5. **Worker exits at boot with `Runtime ... changed packages already imported by the worker`**
|
||||
- A runtime pin in `model-plugin-factory` `runtime/*.yaml` no longer matches `requirements.txt`; the message lists each package as `old -> new`
|
||||
- Set the runtime version in `requirements.txt` and `requirements-local.txt` here and in `sientia-dataops-laborious`, then redeploy
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging by setting the log level:
|
||||
|
||||
Reference in New Issue
Block a user