LiteTopK: 次元の呪いを活用した長文脈スパースアテンション向け融合インデクサー・TopKカーネルLiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention

Long-context language models have become a central focus of machine learning research, yet the computational cost of attention grows quadratically with sequence length, making inference over tens or hundreds of thousands of tokens expensive. A new paper titled LiteTopK proposes a GPU kernel design for sparse attention that targets one of the practical bottlenecks in this setting: the overhead of deciding which past tokens each query should attend to.

Sparse attention is one of the leading strategies for reducing long-context costs. Rather than computing attention over every previous token, these methods estimate which keys are most relevant to a given query and restrict the expensive attention computation to a small subset. This generally involves two stages. First, an indexer produces cheap relevance scores between a query and candidate keys, often using compressed representations, block-level summaries, or clustering. Second, a TopK selection picks the highest-scoring candidates, and only those are forwarded to the full attention kernel. In many implementations, the indexer and the TopK step run as separate operations, which means intermediate scores must be written to and read back from GPU memory.

LiteTopK's central contribution, according to the summary, is fusing the indexer and the TopK selection into a single GPU kernel. Kernel fusion is a well-established optimization: FlashAttention popularized it for exact attention by keeping intermediate results in fast on-chip memory rather than materializing large matrices in slower high-bandwidth memory. LiteTopK appears to apply a similar philosophy to the selection phase of sparse attention, cutting the memory traffic and kernel-launch overhead that accumulate when the two stages execute independently. Because token selection happens repeatedly during autoregressive decoding, even modest per-step savings can compound across a long generation.

The most distinctive element of the work is how it treats the so-called curse of dimensionality. In high-dimensional spaces, a well-documented statistical phenomenon causes pairwise distances between points to concentrate, so the gap between the nearest and farthest candidate shrinks in relative terms as dimensionality grows. This is usually cited as an obstacle for nearest-neighbor search, since it makes close points harder to distinguish from distant ones. LiteTopK instead reframes the property as an asset, apparently using the predictable statistical structure of high-dimensional score distributions to make the TopK step cheaper or more approximate without materially changing which tokens are ultimately selected. If that holds across settings, it suggests the selection can rely on the regularity of the distribution rather than exhaustively comparing every score.

The work sits within a fast-moving line of research on efficient long-context inference. Methods such as Native Sparse Attention, MoBA, Quest, and H2O each propose different ways to prune or prioritize the key-value cache, and several major model developers have shipped sparse or hardware-aware attention variants in production systems. A recurring theme is that the KV cache, not raw compute, is often the binding constraint at long sequence lengths, because it grows linearly with context and must be streamed from memory at every decoding step. Techniques that reduce how much of that cache is touched per token directly ease memory-bandwidth pressure, which is frequently the limiting factor on modern accelerators.

For practitioners, the significance of a fused indexer-and-TopK kernel is largely about closing the gap between theoretical and realized speedups. Sparse attention promises to reduce the number of key comparisons, but if the selection machinery itself introduces overhead through extra memory round-trips or poor GPU occupancy, real-world gains can fall short of the analysis on paper. Consolidating the pipeline into one kernel is a direct attempt to reduce that overhead.

As with many systems-oriented papers, the practical value will depend on details that a summary cannot convey: the accuracy trade-offs of the approximation, how well the approach generalizes across model sizes and context lengths, hardware assumptions, and whether the reported efficiency gains hold on standard long-context benchmarks. Readers evaluating the method would be wise to examine those measurements directly. Still, the underlying idea, turning a statistical property usually viewed as a liability into a lever for efficiency, is a notable framing for the ongoing effort to make long-context inference more affordable.