LLM の性能は prefill と decode で決まり方が違うThis article explains that LLM inference performance is governed by…

When evaluating how fast a large language model runs, it is tempting to reduce the answer to a single factor: a faster GPU, a smaller quantized model, or the sheer number of matrix multiplications involved. A recent write-up on Qiita argues that this framing is incomplete, because LLM inference is not one uniform workload but two distinct phases, prefill and decode, each governed by a different bottleneck. Understanding that split matters for anyone deploying models locally or tuning inference costs, since an optimization that helps one phase may do little for the other.

The prefill phase processes the entire input prompt at once. Because all input tokens are available simultaneously, the hardware can handle them in parallel through large matrix-matrix multiplications. This makes prefill compute-bound: the limiting factor is usually the raw arithmetic throughput of the GPU. Prefill largely determines the "time to first token," the delay a user perceives before any output appears, and its cost scales with prompt length.

The decode phase is different. Here the model generates output one token at a time, and each new token depends on the ones before it. That autoregressive structure turns the workload into a series of matrix-vector multiplications with low arithmetic intensity. For every token produced, the system must read the model's weights from memory, so decode is typically memory-bandwidth bound rather than compute-bound. This is why decode speed often tracks memory bandwidth more closely than peak FLOPS, and why it shapes the inter-token latency and sustained throughput of a running model.

This distinction explains why blanket claims about speed can mislead. Quantization, for instance, reduces the number of bytes that must be moved per token, which tends to help the memory-bound decode phase more directly than it helps compute-bound prefill. A high-FLOPS accelerator with limited memory bandwidth may accelerate prefill while leaving decode relatively unchanged. The phase that dominates a given workload therefore depends on the ratio of prompt length to generated length: long prompts with short answers lean on prefill, while short prompts with long generations are dominated by decode.

A key enabling concept behind this behavior is the KV cache. During attention, the model stores the key and value tensors it has already computed so that each new decode step does not recompute the full history. This avoids repeating prefill-like work at every step, but it also consumes memory that grows with sequence length and batch size, adding pressure on the same memory system that already constrains decode. Managing that cache efficiently is a central concern in production serving.

The roofline model offers a useful mental picture here. It plots achievable performance against arithmetic intensity, showing where a workload transitions from being limited by memory bandwidth to being limited by compute. Prefill and decode tend to sit on opposite sides of that curve, which is another way of framing why they respond differently to hardware and software changes.

These ideas are reflected in the design of modern inference engines. Systems such as vLLM, NVIDIA's TensorRT-LLM, and Hugging Face's Text Generation Inference use techniques like continuous batching to keep the GPU busy across many concurrent requests, improving decode throughput without hurting latency too much. PagedAttention, introduced with vLLM, manages KV cache memory more like virtual memory to reduce waste. Chunked prefill schedules long prompts alongside ongoing decode work, and speculative decoding uses a smaller draft model to propose tokens that the main model verifies, aiming to raise the effective token rate. Lightweight local runtimes such as llama.cpp expose similar trade-offs on consumer hardware.

For practitioners, the practical takeaway is that inference tuning should start by identifying which phase dominates the target workload, then measuring the right metric: time to first token for prefill-heavy cases, tokens per second and inter-token latency for decode-heavy ones. Benchmarks that report only a single number, or that assume a fixed prompt-to-output ratio, can obscure real behavior. As local LLM deployment continues to spread, this phase-aware view of performance is likely to remain a prerequisite for making sensible hardware, quantization, and serving decisions.