LLMにおけるアライメントの痕跡を計測・局在化・除去する研究 Measuring, Localizing, and Ablating Alignment Signatures in LLMs

Aligned large language models have a tell. The over-polite refusals, the hedged phrasing, the boilerplate safety disclaimers — users who spend time with these systems quickly learn to recognize a certain AI-ness in their outputs. But exactly where in a model's internals does this style live, and how does post-training give rise to it? A new paper from arXiv (2605.30526) takes a rigorous look at these questions.

The researchers introduce the concept of an 'alignment signature' — measurable traces left in a model's internal representations by post-training procedures such as RLHF and supervised fine-tuning. Their methodology involves comparing the activations of aligned models against their base-model counterparts across layers and attention heads, then using targeted ablation to surgically remove those signature components and observe the behavioral effects.

The core findings suggest that alignment-related features are not uniformly distributed throughout a model but are concentrated in specific layers. By identifying these locations, the team demonstrates that it is possible to attenuate the recognizable AI-like stylistic tendencies without causing a corresponding degradation in general capability — a result with both practical and theoretical implications.

This work sits within a rapidly growing field of mechanistic interpretability research. Groups at Anthropic, EleutherAI, and various academic labs have pursued similar goals: understanding how abstract concepts like 'refusal', 'safety', or 'politeness' are encoded as geometric structures in high-dimensional activation spaces. Techniques like activation patching, probing classifiers, and sparse autoencoders have all been brought to bear on questions of this kind. Localizing alignment signatures is a natural extension of that agenda.

The dual-use dimension of this research deserves acknowledgment. Demonstrating that alignment signatures can be identified and ablated is, almost by definition, a step toward showing how alignment might be bypassed. The authors appear aware of this tension, framing the work as a defensive contribution — one that gives alignment researchers and red-teamers a clearer picture of where the weaknesses in current post-training approaches lie. The argument is that you cannot harden what you cannot see.

For practitioners, the implications are notable. If alignment is encoded in predictable, localizable ways, that raises questions about the robustness of current RLHF pipelines. A sufficiently motivated adversary with model access could, in principle, use techniques like those described here to strip away safety-relevant behaviors. At the same time, the same localization methods could inform more robust training strategies that spread alignment-critical features more diffusely, making them harder to ablate.

The paper is a reminder that interpretability and safety are deeply intertwined research programs. Progress in one tends to enable progress in the other — for better and for worse. As mechanistic interpretability matures, the field will need clear norms around responsible disclosure of findings that have obvious offensive applications. This paper likely won't be the last to raise that question.