speculative decoding×prefix cachingの罠：組み合わせで遅くなるケースCombining MTP speculative decoding with prefix caching in vLLM can halve cache…

Combining two well-known inference optimizations does not always yield the sum of their benefits, and a recent report against vLLM illustrates why this matters for anyone tuning latency. Engineers commonly assume that enabling both speculative decoding and prefix caching will compound the speedups, but in practice the two features can interfere, and in at least one documented case they degrade rather than improve latency.

The specific problem surfaces in vLLM's GitHub issue tracker, where a user reports that turning on MTP (Multi-Token Prediction) speculative decoding while prefix caching is active causes the effective cache hit length to fall from 32,768 tokens to 16,384 tokens. That halving of cached context translates directly into a significant increase in TTFT (Time To First Token), the latency between a request arriving and the first output token being produced. In other words, the configuration that was supposed to make the system faster ends up making the initial response slower.

To understand why, it helps to review what each technique does. vLLM is a high-throughput serving engine best known for PagedAttention, which manages the key-value (KV) cache in fixed-size blocks similar to virtual memory paging. Prefix caching builds on this by storing the KV cache for prompt prefixes shared across requests. When a new request begins with the same tokens as a cached prefix—a system prompt, a shared document, or a long conversation history—the engine can reuse that computation instead of recomputing attention from scratch. The longer the reusable prefix, the less prefill work is required, and the lower the TTFT.

Speculative decoding attacks a different bottleneck. Instead of generating one token per forward pass of the large model, it uses a cheaper mechanism—a draft model or, in the MTP case, additional prediction heads that propose several future tokens at once—to guess multiple tokens ahead. The large model then verifies those guesses in a single pass, accepting the ones that match and discarding the rest. When guesses are accurate, throughput rises because more tokens are confirmed per expensive forward pass. MTP is a variant popularized by models such as DeepSeek-V3, where the architecture itself is trained to predict several tokens per step.

The friction appears to arise from how these two systems share and account for the KV cache. Speculative decoding generates and verifies speculative tokens that may later be rejected, which affects how blocks are allocated, aligned, and committed. Prefix caching, meanwhile, depends on stable, block-aligned prefixes to match and reuse. If the speculative path changes block boundaries or reserves capacity for draft tokens, the length of prefix the cache can cleanly match may be reduced—here, apparently from a full 32,768-token window down to half that. The result is that fewer tokens are served from cache, more prefill must be recomputed, and TTFT climbs.

This kind of interaction is a useful reminder that inference optimizations are rarely orthogonal. Both features compete for the same finite GPU memory and the same KV-cache block accounting, so enabling one can shrink the headroom or break the assumptions the other relies on. The benefit each delivers is also workload-dependent: prefix caching pays off most with long shared prefixes and repeated prompts, while speculative decoding pays off most when draft acceptance rates are high. A gain in one dimension does not guarantee a net win when both are active.

For practitioners, the practical takeaway is to measure rather than assume. Benchmarking TTFT, cache hit rate, and end-to-end throughput with each optimization individually and in combination—on representative traffic—is the only reliable way to know whether the pairing helps. Because vLLM is under active development and the behavior is tracked as an open issue, the specifics are likely to change as the project evolves, and a fix or configuration workaround may narrow the gap. Until then, teams running long-context workloads that lean heavily on prefix caching should be cautious about assuming MTP speculative decoding will stack cleanly on top.

The broader lesson extends beyond this single bug. As serving stacks accumulate more clever features—chunked prefill, continuous batching, quantization, and speculative methods among them—the surface area for subtle, negative interactions grows. Treating each optimization as an independent multiplier is convenient but risky; validating the combined behavior on real prompts remains the safer engineering discipline.