べき乗変換と符号保存スコア集約による大規模言語モデルの構造的プルーニング Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

Large language models keep growing, and with them the cost of running inference, so techniques that shrink a trained network without retraining it from scratch have become an active research front. A new paper posted to arXiv proposes a structured pruning method for LLMs that combines a power transformation of importance scores with what the authors call sign-preserving score aggregation, along with adaptive feature retention. The stated goal is to remove parameters more aggressively while keeping accuracy degradation smaller than existing approaches, which matters because memory footprint, latency, and energy consumption remain the practical barriers to deploying capable models on constrained hardware.

Pruning generally comes in two flavors, and the distinction is important for understanding what this work targets. Unstructured pruning zeroes out individual weights, producing sparse matrices that can be very accurate but that mainstream GPUs and inference runtimes struggle to accelerate without specialized kernels. Structured pruning instead removes whole components such as attention heads, neurons, channels, or intermediate dimensions, yielding a smaller dense model that runs faster on standard hardware. The trade-off is that removing coarse-grained structures tends to hurt accuracy more, so the central challenge is deciding which structures are least important and can be dropped with minimal impact.

That decision hinges on importance scoring, and this is where the paper's contributions sit. Most pruning schemes assign each weight or unit a saliency score, often derived from weight magnitudes, activations, or gradient-based sensitivity, and then aggregate those scores across a dimension to rank entire structures for removal. The authors argue that naive aggregation can be distorted by the shape of the score distribution and by cancellation between positive and negative contributions. Applying a power transformation reshapes the distribution of scores before aggregation, which appears intended to reduce the dominance of a few outliers and produce more reliable rankings. Sign-preserving aggregation is meant to retain information about the direction of each contribution rather than collapsing everything into magnitudes, so that units whose effects would otherwise cancel out are not misjudged as unimportant. Adaptive feature retention rounds out the approach by keeping a variable set of features rather than pruning uniformly across the network.

The method belongs to a lineage of recent LLM-specific pruning work. LLM-Pruner popularized gradient-based structured pruning followed by lightweight recovery fine-tuning, while SliceGPT reduced the embedding dimension through a transformation of the weight matrices. On the unstructured and semi-structured side, SparseGPT and Wanda demonstrated that one-shot pruning using calibration data could reach high sparsity without full retraining, with Wanda in particular showing that combining weight magnitude with input activation norms is a strong and cheap saliency signal. The new paper's emphasis on how scores are transformed and combined can be read as a refinement of this scoring-and-aggregation step rather than a wholesale departure from it.

It is also worth situating pruning within the broader model-compression toolkit, since practitioners rarely rely on a single technique. Quantization, which lowers the numerical precision of weights and activations to 8-bit, 4-bit, or lower, is often the first lever pulled because it is well supported by libraries such as GPTQ, AWQ, and bitsandbytes. Knowledge distillation trains a smaller student model to imitate a larger teacher. These methods are largely complementary to pruning, and a compressed deployment pipeline may stack them, for example by pruning structurally and then quantizing the survivor. Structured pruning is attractive precisely because it produces a smaller dense model that composes cleanly with these other steps.

As with most preprints in this area, the claims should be read with appropriate caution until they are independently reproduced. Pruning results are sensitive to the choice of calibration data, the sparsity level, the model family, and whether any recovery fine-tuning is applied, and gains reported on perplexity or a handful of benchmarks do not always transfer to downstream tasks or to larger models. The reported combination of power transformation and sign-preserving aggregation is likely most valuable at moderate to high pruning ratios, where ranking errors are costliest. Readers interested in the specifics of the transformation, the calibration setup, and the evaluated model sizes should consult the paper directly, as those details determine how broadly the improvements are likely to hold.