Notes · Field note ·
The engine that said it was healthy
A health check tells you the door opens. It doesn't tell you anyone's home.
At 00:37 one night, a local model server's work loop stopped. Clients kept sending requests; 83 of them timed out. The GPU sat at 0 percent.
The server's metrics stopped answering. Its health endpoint kept returning 200 OK, so neither my router nor the keep-alive task noticed anything was wrong.
Recovery was worse. The clean stop command needs the idle gauges, which weren't answering. The service stop timed out after three minutes, the process held 77.7 GB of GPU memory even after a kill, and only shutting down the whole Linux subsystem freed the card.
What changed, that same night
- By 1:45 AM the router judged liveness from live metrics instead: asked every 15 seconds, with an 8-second timeout.
- One miss marks a lane suspect; two in a row mark it hung, and it's taken out of service automatically.
- A test now reproduces the hung work loop, and it's declared hung after two strikes.
Check the thing you care about, not the thing that's easy to check.