gitea_admin/sientia-dataops-model-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(worker): set app_up only after the runtime version guard (SIENTIAPDE-2214)

Browse Source 
The probes only check app_up, which was set to 1 when the metrics server started,
before install_runtime. A pod that would fail the guard could turn Ready first, so a
rollout would remove the old pod. app_up now goes to 1 right after the guard.

Also: copy sys.modules keys before iterating, raise the sientia_model floor to the
runtime's >=0.14.0, pin the same runtime versions in requirements-local.txt, test
that the snapshot is taken before the install and the real importlib.metadata path,
and document the boot failure in README and CLAUDE.md.
This commit is contained in:
PedroHMCosme
2026-10-22 12:17:12 -03:00
parent 0f5e4a5cef
commit dfbdd4b605
6 changed files with 74 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
model_manager/worker/worker.py
View File

@@ -79,7 +79,8 @@ def _get_runtime(runtime: str | None) -> str:
def _loaded_distribution_versions() -> dict[str, str]:
"""Version on disk of every installed distribution whose top-level module is imported."""
loaded = {name.partition('.')[0] for name in sys.modules}
# list() copies the keys: another thread importing while we iterate would raise RuntimeError.
loaded = {name.partition('.')[0] for name in list(sys.modules)}
return {
dist: importlib.metadata.version(dist)
for module, dists in importlib.metadata.packages_distributions().items()
@@ -114,7 +115,8 @@ async def main():
1. Sets up logging and metadata
2. Starts Prometheus metrics server
3. Initializes notification handler, then installs the runtime; exits if the install
changed the version of a package the worker already imported
changed the version of a package the worker already imported, and only then sets
``app_up`` (what the probes check) to 1
4. Creates and configures activities
5. Starts Temporal client and workers
6. Manages worker lifecycle and graceful shutdown
@@ -188,6 +190,10 @@ async def main():
metrics.APP_UP.labels(pod_id=POD_ID).set(0)
sys.exit(1)
# The probes only check app_up, so it goes to 1 after the guard: a pod that would fail it
# never turns Ready, and a rollout keeps the old pod.
metrics.APP_UP.labels(pod_id=POD_ID).set(1)
activities = Activities(
postgres_config=build_postgres_config(),
mlflow_config=build_mlflow_config(),
@@ -274,8 +280,8 @@ def start_prometheus_server(logger: SientiaLogger, metadata: dict[str, str | Non
"""
Starts the Prometheus metrics server for monitoring and observability.
This function initializes the Prometheus HTTP server on the configured port
and sets the application health metric to indicate the service is running.
This function initializes the Prometheus HTTP server on the configured port.
``main`` sets ``app_up`` to 1 later, once the runtime install passed the version guard.
The server exposes metrics that can be scraped by Prometheus for monitoring
the health and performance of the Model Manager worker.
@@ -291,7 +297,6 @@ def start_prometheus_server(logger: SientiaLogger, metadata: dict[str, str | Non
port = int(os.getenv('HTTP_METRICS_PORT', 9090))
start_http_server(port)
logger.custom_info(f'Prometheus server initialized on port {port}.', metadata)
metrics.APP_UP.labels(pod_id=POD_ID).set(1) # Mark app as UP
except Exception as e: # noqa: BLE001
logger.custom_critical(f'Failed to start Prometheus server: {e}', metadata)
os._exit(1)