一般化レイリー商最適化による基盤モデルの能力保持型ファインチューニング Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization

One of the enduring challenges in machine learning is adapting large pretrained models to specialized tasks without erasing the general capabilities they acquired during pretraining. A new paper on arXiv, "Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization," takes an analytically grounded approach to this problem, drawing on classical linear algebra to reformulate the finetuning objective itself.

Finetuning has become the dominant paradigm for deploying foundation models across natural language processing, computer vision, and speech. Yet standard gradient-based finetuning optimizes purely for target-task performance, which can quietly degrade capabilities the model was never explicitly asked to retain. This phenomenon — variously described as catastrophic forgetting or capability collapse — poses real risks in production settings where a model is expected to generalize beyond its narrow finetuning distribution.

The generalized Rayleigh quotient is a concept borrowed from numerical linear algebra and quantum mechanics, expressing the ratio of a vector's energy under two different quadratic forms. By incorporating this quantity into the optimization objective, the authors appear to create a principled mechanism for balancing task adaptation against the preservation of pretrained capability subspaces. Rather than treating capability retention as a post-hoc regularization afterthought, the framework embeds it directly into how parameter updates are computed.

This places the work in conversation with a rich ecosystem of parameter-efficient finetuning (PEFT) research. Methods like LoRA, Adapter layers, DoRA, and IA³ address related concerns by constraining updates to low-rank subspaces, reducing both computational cost and the risk of overwriting pretrained structure. The Rayleigh-quotient framing offers a different geometric lens: rather than implicitly limiting update magnitude through rank constraints, it may explicitly define which directions in weight space are off-limits via an eigenvalue-derived criterion.

The timing of this work reflects broader anxieties in the field. As foundation models grow larger and more expensive to train, the cost of capability degradation during finetuning increases proportionally. Enterprises deploying models across multiple use cases need assurance that domain-specific tuning won't silently undermine performance on tasks they care about but didn't include in the finetuning data. Safety-conscious deployments face similar concerns — alignment finetuning should not inadvertently compromise factual accuracy or reasoning robustness.

It remains to be seen whether the proposed method scales gracefully to very large models or integrates naturally with existing PEFT toolchains like Hugging Face's PEFT library. If the mathematical overhead is tractable, the approach could complement rather than compete with low-rank methods. Whether or not it achieves wide adoption, the paper represents a thoughtful attempt to bring analytical rigor to a problem that most practitioners currently handle with heuristics and hyperparameter tuning.