ブラックボックス LLM 蒸留における有界行動不可識別性 Bounded Behavioral Indistinguishability for Black-Box LLM Distillation

One of the most persistent blind spots in LLM research has been the lack of a rigorous definition for what it means for a student model to successfully imitate a teacher. Black-box distillation — where the student learns solely from the teacher's outputs, with no access to weights or logits — has been evaluated almost exclusively through output-matching metrics like semantic similarity or BLEU scores. A new paper proposes a more principled alternative: bounded behavioral indistinguishability.

The core argument is that output-matching metrics are inherently local. They assess how well a student responds to a given set of prompts but say nothing about whether the student's overall behavior is indistinguishable from the teacher's across the full input distribution. Long-tail queries and adversarial prompts are precisely where the two can diverge most sharply — and where the gap matters most for downstream safety and reliability.

The framework introduced in the paper borrows conceptually from cryptographic notions of computational indistinguishability. Rather than asking whether individual outputs match, it asks whether a probabilistic distinguisher — an algorithm that sees responses from both models — can tell them apart with probability meaningfully above chance. By bounding the advantage such a distinguisher can achieve under a given input distribution, the framework provides a formal, quantifiable notion of distillation success.

This kind of formalization arrives at a meaningful moment. The past two years have seen an explosion of black-box distillation work targeting commercial APIs from OpenAI, Anthropic, Google, and others. Several high-profile model releases have been accused of being distilled from GPT-4 or Claude without authorization, prompting heated debates about terms of service and intellectual property. What has been missing from those debates is a precise language for describing how similar a model actually is. The bounded behavioral indistinguishability framework could, in principle, provide exactly that — though its application to legal or policy contexts would require significant additional work.

There is also a natural connection to the growing literature on model fingerprinting and watermarking. If a teacher model embeds detectable statistical patterns in its outputs, the question of whether a student inherits those patterns is essentially a question about behavioral indistinguishability — and about how tight the bound is. The two research threads seem likely to converge as the field matures.

From a practical standpoint, the framework raises interesting questions about how to operationalize the distinguisher. Choosing the right input distribution is non-trivial: a student might be indistinguishable under typical user queries but highly distinguishable under carefully crafted adversarial inputs. How the paper handles this tradeoff, and whether it proposes empirically tractable approximations, will likely determine how widely adopted the framework becomes.

Overall, this work represents a step toward placing LLM distillation evaluation on firmer theoretical ground. Whether the community moves toward adopting formal indistinguishability metrics alongside existing benchmarks remains to be seen, but the conceptual contribution — reframing distillation success as a distributional, probabilistic property rather than a point-wise one — seems likely to influence how the field thinks about model imitation for some time.