Google TPUでLLM推論を3倍高速化、拡散型投機デコードを採用 Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding

Google has unveiled a new diffusion-inspired speculative decoding technique that delivers up to 3X faster LLM inference on its TPU hardware. The work targets one of the most pressing bottlenecks in production generative AI: the cost and latency of serving large autoregressive models at scale.

Standard autoregressive decoding generates tokens one at a time, leaving much of the parallel compute capacity of modern accelerators idle. Speculative decoding addresses this by using a lightweight draft model to propose several future tokens, which the larger target model then verifies in a single parallel forward pass. When the drafts are accurate, multiple tokens are accepted per step, multiplying effective throughput without changing the output distribution.

Google's twist borrows from diffusion models, which iteratively refine noisy candidates into coherent outputs. Rather than producing a strict left-to-right draft, the system refines a block of candidate tokens in parallel, in a manner well suited to the matrix-heavy systolic array architecture of TPUs. The reported 3X speedup suggests meaningful gains in tokens-per-second and, consequently, lower serving costs for high-throughput workloads such as chat assistants, code completion, and batch summarization.

Speculative decoding has become one of the most active areas of LLM systems research. Variants such as Medusa, EAGLE, Lookahead Decoding, and self-speculative approaches have appeared from academic labs and industry players including Meta, Together AI, and Anthropic. Each tries to balance draft quality, verification overhead, and hardware utilization. Diffusion-style parallel refinement is a natural fit for accelerators that prefer dense, predictable compute patterns, and TPUs in particular benefit from workloads that minimize sequential dependencies.

The announcement also highlights Google's continued strategy of co-designing models, software, and silicon. While much of the open ecosystem is optimized for NVIDIA GPUs and CUDA, Google has invested heavily in JAX, XLA, and Pallas to extract performance from TPUs. Techniques like this one likely underpin the serving infrastructure for Gemini and may eventually be exposed through Vertex AI or the Gemini API, although Google has not detailed a specific rollout. For developers evaluating inference platforms, the result is another data point suggesting TPUs can be competitive on cost-per-token for large-scale deployments, particularly when paired with Google's proprietary optimizations.

It remains to be seen how the approach generalizes across model sizes, context lengths, and decoding settings such as temperature and beam search. Independent benchmarks and, ideally, a published technical report would help the community assess where the 3X figure holds and where trade-offs emerge. Still, the direction reinforces a broader industry trend: the next wave of LLM efficiency gains will come not just from smaller models or quantization, but from rethinking the decoding loop itself.