待ち時間を予測してLLM推論を振り分ける、llm-dの予測レイテンシ・ルーティングllm-d's predicted-latency routing automatically balances KV-cache locality…

Routing requests across a fleet of GPU servers is one of the less glamorous but more consequential problems in large language model serving, and llm-d is proposing a way to make it self-tuning. The project's predicted-latency routing tries to resolve the long-standing tension between reusing cached computation and spreading load evenly, without asking operators to hand-tune the balance between the two.

The underlying difficulty comes from how transformer inference works. When a model processes a prompt, it builds a key-value (KV) cache that can be reused if a later request shares the same prefix, letting the server skip part of the expensive prefill stage. That creates a strong incentive to send requests with identical prefixes to the same server. But if a scheduler leans too hard on this cache locality, the popular server fills up, its queue grows, and tail latency spikes. Push in the opposite direction and spread traffic evenly, and the cache goes cold, forcing servers to redo prefill work they could have avoided.

This is essentially a tug-of-war between cache locality and load distribution. In production, prompt lengths and output lengths can swing by orders of magnitude within a few hours, so any fixed weighting that favors one side over the other will eventually break down as traffic shifts. That is why operators historically end up babysitting these knobs.

llm-d's answer is to stop expressing the trade-off as a static weight and instead convert both concerns into a single, comparable quantity: predicted latency. Rather than scoring a server on "how good is its cache hit" and "how busy is it" separately and blending those scores by hand, the scheduler estimates how long a given request would actually take on each candidate server and routes to whichever minimizes that predicted wait. A cache hit is only worth chasing when the target server is not so congested that queueing wipes out the savings, and the prediction captures that directly.

In practice this appears to combine an estimate of prefill and decode cost, which depends on how much of the prompt is already cached versus must be computed, with an estimate of queueing delay derived from each server's current load. Because both are expressed in units of time, they become directly comparable, and the system can adapt as conditions change instead of relying on a weight that was correct only for yesterday's traffic. The stated goal is to keep inference latency stable under variable production load while eliminating manual tuning.

It helps to place this in context. llm-d is a Kubernetes-native distributed inference stack built around vLLM, developed with contributions from across the industry, and aimed at serving models