0次選択から2次判定へ:組合せ強化でフロンティアLLMの構成的失敗を露呈 From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs

Standard LLM benchmarks built around multiple-choice questions are showing clear signs of saturation, making it increasingly difficult to distinguish surface pattern matching from genuine reasoning. This paper proposes a methodology called combinatorial hardening, which systematically escalates evaluation difficulty and reveals that frontier models fail at compositional judgment even when they reliably know the underlying facts.

The authors organize evaluation into three orders. A 0-order task is the familiar multiple-choice format, where a model selects one answer from a candidate set. A 1-order task asks the model to judge the truth of a single proposition, removing the elimination heuristics that multiple-choice formats invite. A 2-order task presents combinations of propositions and asks the model to judge their joint consistency or truth, forcing actual integration across statements rather than isolated recall.

The central empirical finding is a sharp degradation as order increases. Models that score highly on 0-order versions of a benchmark can collapse on the 2-order variant covering the same knowledge. This gap is not easily explained by missing facts, since the 1-order results indicate the atomic propositions are largely known. Instead, it points to a compositional weakness: frontier LLMs struggle to combine pieces of knowledge they individually possess into a coherent verdict.

This line of work fits into a broader ecosystem of evaluations designed to resist shortcut solutions, including GSM-Symbolic, ARC-AGI, BIG-Bench Hard, and various perturbation-based probes. A recurring theme across these efforts is that headline scores on legacy benchmarks overstate model competence because the formats themselves leak information. The combinatorial hardening approach is notable for being format-agnostic in spirit; it can in principle be applied on top of existing question banks by reformulating them as truth judgments over combinations.

The practical implications matter for anyone deploying LLMs in agentic or retrieval-augmented settings. Real workflows rarely involve picking among neatly enumerated answers. They involve gathering several facts, possibly from tools or documents, and arriving at a combined conclusion. If models retrieve correctly yet integrate poorly, downstream reliability suffers in ways that 0-order accuracy metrics will not catch. This may partly explain why agent benchmarks often look worse than chat benchmarks for the same underlying model.

On the training side, the results suggest that scaling pretraining alone is unlikely to close the compositional gap. Approaches that explicitly target multi-step verification, such as process reward models, self-consistency, deliberate chain-of-thought training, and synthetic data emphasizing compositional structure, are plausibly more effective levers. Reasoning-focused models in the o-series and similar systems have shown gains on related tasks, though it remains to be seen whether they fully overcome the 2-order penalty reported here.

Readers should note that the arXiv identifier formatting in the source appears unusual, so submission date and version details are best confirmed against the canonical record. Regardless of metadata particulars, the methodological contribution stands on its own: by moving the evaluation lens from selection to combinatorial judgment, the paper provides a sharper instrument for diagnosing where current frontier models actually break down.