BitsMoE: スペクトルエネルギーを活用したMoE LLMの効率的ビット割り当て量子化 BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

Mixture-of-Experts (MoE) architectures have become one of the most compelling strategies for scaling large language models without proportionally scaling inference compute. By activating only a sparse subset of expert networks per token, MoE models reduce per-token FLOPs significantly. The catch, however, is that all expert weights must reside in memory simultaneously, making MoE models notoriously memory-hungry at deployment time. BitsMoE directly targets this tension by introducing a quantization framework that allocates bit widths intelligently rather than uniformly across layers.

The central idea behind BitsMoE is to use the spectral energy distribution of each weight matrix — derived via singular value decomposition — as a proxy for its informational importance. Layers whose weight matrices exhibit high spectral energy concentration are assigned more bits, preserving their representational capacity. Conversely, layers with flatter spectral profiles receive fewer bits, trading off precision where the model can afford it. This non-uniform allocation stands in contrast to conventional post-training quantization schemes like GPTQ or AWQ, which tend to apply fixed bit widths globally or rely on activation-based sensitivity metrics rather than the intrinsic geometry of the weight matrices themselves.

The timing of this work reflects the rapid rise of MoE as a mainstream architecture. Models like Mixtral 8x7B and 8x22B from Mistral AI, DeepSeek's MoE variants, and Google's Gemini-family mixture architectures have all pushed MoE into production workloads. Existing quantization tooling, however, was largely developed with dense transformer models in mind. MoE-specific challenges — including the high variance in expert weight distributions and the need to handle gating mechanisms gracefully — create a genuine gap that BitsMoE appears designed to address.

The use of spectral energy as a sensitivity signal is intellectually adjacent to low-rank approximation methods like LoRA, which similarly leverage singular value structure to identify which components of a weight matrix carry the most information. BitsMoE extends this intuition into the quantization domain, asking not "which directions to keep" but "how precisely to represent each layer." This framing could make it complementary to existing compression pipelines rather than a replacement for them.

For practitioners, the practical stakes are considerable. Running a high-quality MoE model on a single consumer GPU or on-device hardware requires aggressive memory reduction without catastrophic quality loss — a bar that uniform low-bit quantization often struggles to clear. If BitsMoE's spectral guidance can reliably identify where bit budget is best spent, it could make models like Mixtral or DeepSeek-MoE viable on hardware that currently cannot fit them. Independent benchmarking and community reproduction will be essential to validate these claims, but the approach is technically well-motivated and worth close attention from the quantization research community.