vLLMの-dcpが長文脈のKVキャッシュ重複を消して3倍速くするvLLM's -dcp option eliminates redundant KV cache memory caused by tensor…

Running large language models with very long context windows has become one of the harder problems in production inference, and a configuration flag in vLLM called -dcp is aimed squarely at it. According to a technical write-up on Qiita, the option eliminates redundant key-value (KV) cache memory that tensor parallelism introduces in long-context deployments, and it can deliver up to a threefold increase in throughput by reclaiming GPU memory that was previously wasted on duplicated cache entries.

The problem it targets is one that operators of multi-GPU servers may recognize. When a long-context model is spread across eight GPUs using tensor parallelism, the system can suddenly stop accepting new requests even though compute capacity remains available. The bottleneck is not arithmetic throughput but memory: the KV cache fills up, and once it is full there is no room to admit additional concurrent conversations. Crucially, the excerpt notes that the memory pressure comes from duplication rather than raw computation, and that the duplication was, given the model's structure, nearly impossible to avoid.

To understand why, it helps to recall how tensor parallelism and modern attention layouts interact. Tensor parallelism splits a model's weights and attention heads across GPUs so that each device handles a slice of the work. Many recent models, however, use grouped-query attention (GQA) or multi-query attention (MQA), which deliberately reduce the number of key-value heads to save memory. When the number of KV heads is smaller than the number of GPUs in the tensor-parallel group, those heads cannot be divided cleanly, so the KV cache for them is replicated on every GPU. In an eight-way split of a model with only one or two KV heads, the same cache entries end up copied many times over, consuming memory that scales poorly as context length grows.

The -dcp flag, which corresponds to decode-time context parallelism, changes the axis along which the cache is partitioned. Instead of replicating KV heads across devices, it shards the cache along the sequence (context) dimension, so each GPU stores a distinct portion of the tokens rather than a duplicate copy. The result is that the aggregate KV cache capacity grows with the number of GPUs instead of being pinned to the memory of a single device. Freeing that duplicated memory lets the server hold more concurrent sequences or longer histories, which is where the reported throughput gains of up to three times come from. The improvement appears most pronounced in long-context, memory-bound scenarios; gains for short prompts are likely to be smaller, since duplication matters less when the cache is not the limiting factor.

This work sits within a broader effort across the inference ecosystem to make the KV cache cheaper to store and move. vLLM itself is built around PagedAttention, which manages the cache in non-contiguous blocks to reduce fragmentation, and it already supports prefix caching to reuse shared prompt segments. Other systems, including SGLang, TensorRT-LLM, and various serving stacks, have pursued complementary ideas such as KV cache quantization, offloading to CPU or NVMe, and different forms of context and sequence parallelism. Context parallelism in particular has gained attention as models push toward hundred-thousand-token and longer windows, where the cache rather than the weights becomes the dominant memory cost.

For practitioners, the practical takeaway is that the choice of parallelism strategy interacts with a model's attention design in ways that can quietly cap concurrency. Operators who see GPUs sitting idle while request queues back up may be hitting KV cache exhaustion rather than a compute wall, and options like -dcp are intended to address exactly that pattern. As always with such features, the benefits depend on the specific model, GPU count, and workload, and users should validate the throughput and latency trade-offs against their own traffic before relying on the reported figures. Still, the change reflects a clear direction of travel: as context windows lengthen, efficient partitioning of the KV cache is becoming as important as partitioning the model weights themselves.