VRAMに乗らないMoEをNVMe＋GPU推論で動かす：Hypura/llama.cpp/TurboQuant解説This article explains how to run large MoE models that exceed VRAM capacity by…

Mixture-of-Experts (MoE) architectures have become one of the dominant designs for frontier-scale language models, but their size creates a practical problem: the full set of expert weights often far exceeds the memory available on consumer GPUs. A recent write-up on Zenn examines how to bridge that gap by offloading model layers to NVMe storage while still using a GPU for acceleration, drawing on three tools — Hypura, llama.cpp, and TurboQuant. The topic matters because it suggests a workable route to running very large models on hardware that would otherwise be far too small to hold them.

The reason MoE models are a natural fit for this approach lies in how they compute. A dense model activates every parameter for every token, so its full weight set must be readily accessible. An MoE model, by contrast, routes each token to only a small subset of its experts — often two out of dozens — meaning that only a fraction of the total parameters participate in any single forward pass. This sparse activation lowers the effective bandwidth demand per token relative to the model's total footprint, which is precisely what makes storing the bulk of the weights on slower media more tolerable.

Understanding the approach requires keeping the memory hierarchy in mind. GPU VRAM offers the highest bandwidth, often around a terabyte per second, but consumer cards top out at roughly 8 to 24 GB. System RAM is larger but an order of magnitude slower, and NVMe SSDs are slower still, delivering perhaps 3 to 7 GB/s on PCIe 4.0 drives and up to around 14 GB/s on PCIe 5.0. The strategy described is essentially a tiering exercise: keep the most frequently used tensors, such as attention layers and the router, in VRAM, and stream expert weights from NVMe as they are needed.

llama.cpp is the most established component here. It runs models in the quantized GGUF format and supports partial GPU offload, letting users place a chosen number of layers on the GPU while the remainder stay in CPU memory or are memory-mapped from disk. Its mmap-based loading lets the operating system page weights in on demand, which is what allows a model file larger than RAM to be opened at all. TurboQuant appears in the article as the quantization piece, reducing the model's size so that more of it fits into faster tiers; aggressive quantization to 4-bit or lower is generally what makes multi-hundred-billion-parameter models even approachable on a single machine. Hypura is presented as the tool coordinating the NVMe-plus-GPU offload path, though readers should consult the original for the exact configuration details.

For broader context, this is not an isolated effort. The KTransformers project has pursued similar CPU-GPU hybrid execution aimed specifically at large MoE models such as DeepSeek's, and groups like Unsloth have popularized dynamic quantization schemes that preserve accuracy in sensitive layers while compressing the rest. The common thread is exploiting MoE sparsity, quantization, and the memory hierarchy together, rather than relying on any single trick. The release of open-weight MoE models, including DeepSeek-V3 and various Qwen and Mixtral variants, has given the community concrete targets for this kind of experimentation.

The practical caveats are significant and worth emphasizing. Throughput bound by NVMe reads is typically far lower than a fully VRAM-resident setup, so users should expect single-digit or low tokens-per-second figures in many configurations rather than interactive speeds. Drive endurance, quantization-induced quality loss, and the complexity of tuning how many layers to offload all factor into whether the tradeoff is worthwhile. Performance also depends heavily on how well an implementation predicts and prefetches the experts a given token will need.