層プルーニングLLMの性能崩壊を決定表現遷移から解明 Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions

Layer pruning, the practice of removing entire Transformer blocks from a trained large language model, has become an attractive route to reducing inference cost because it requires no specialized kernels and integrates cleanly with existing serving stacks. However, practitioners routinely observe a sharp cliff: removing one specific layer barely affects quality, while removing another causes downstream tasks to collapse. This paper attempts to explain that cliff in terms of how decision-relevant representations evolve across the network's depth.

The authors track how hidden states at each layer relate to the model's eventual token-level decisions, treating the forward pass as a trajectory that gradually converges on a prediction. Their analysis suggests that early and mid layers do the heavy lifting of forming this decision representation, while later layers act more like refiners. When pruning targets a layer inside this formation phase, the trajectory is broken in a way that subsequent layers cannot recover from, producing the abrupt performance drop seen empirically.

A practical implication is that common importance heuristics, such as activation magnitudes, gradient norms, or block-wise cosine similarity between input and output, may not capture this dynamic. Methods like ShortGPT and LLM-Pruner have proposed such scoring schemes to choose which blocks to drop, and they work well on some models but fail unpredictably on others. The transition-based view offered here could complement those scores by flagging layers whose removal would sever the representational trajectory, even if their local input-output similarity looks high.

The broader context is a crowded compression landscape that includes quantization, distillation, and structured pruning of attention heads or MLP channels. Layer dropping is appealing because it composes naturally with quantization and yields predictable latency gains on commodity GPUs. Yet the present work hints that what is lost when a layer disappears is not merely a slice of compute but a stage in the model's reasoning path, which may explain why simple post-hoc fine-tuning sometimes fails to recover accuracy.

Several directions appear promising as follow-up work. Distillation that explicitly preserves the inter-layer trajectory, rather than just the final logits, could allow more aggressive depth reduction. Inserting lightweight adapters at pruned positions to bridge the representational gap is another plausible remedy, conceptually similar to how LoRA modules can repair quantization damage. It also remains an open question whether the formation-versus-refinement split observed here generalizes across model families, scales, and instruction-tuned variants, or whether it is sensitive to training recipe. Readers should treat the specific layer indices reported as suggestive rather than universal, but the underlying framing, viewing pruning damage as a disruption of decision-representation flow, is a useful lens for anyone building compressed deployment pipelines.