Format Sensitivity Index：トークン制御プロンプトラッパーの堅牢性とLLMベンチマークにおけるスキーマ準拠Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

Large language models are increasingly evaluated and deployed on the assumption that a benchmark score reflects a stable underlying capability. A new paper posted to arXiv challenges that assumption by introducing the Format Sensitivity Index, or FSI, a metric designed to quantify how much a model's outputs shift when the surrounding prompt structure is altered without changing the substantive task. The work matters because if a model's answers swing depending on superficial formatting choices, the reliability of comparative benchmarks and the trustworthiness of production deployments both come into question.

At the center of the paper is the notion of a prompt wrapper: the scaffolding of instructions, delimiters, role tags, system messages, and formatting tokens that surround the actual question or task. In practice, two evaluators can pose the same logical query while wrapping it very differently, for example by using JSON braces versus XML tags, by changing whitespace and separators, or by reordering instruction blocks. The authors argue that these token-level variations are largely semantic no-ops for a human reader, yet they can meaningfully perturb model behavior. FSI is proposed as a way to put a number on that instability, allowing researchers to compare how robust different models are to the same set of wrapper perturbations.

Methodologically, the index appears to work by holding the task content fixed while systematically applying a family of format transformations, then measuring the divergence in outputs across those variants. The paper emphasizes two related but distinct concerns. The first is answer stability, meaning whether the model reaches the same conclusion regardless of packaging. The second is schema compliance, meaning whether the model reliably returns output in the requested structure, such as valid JSON conforming to a specified schema. A model might remain correct on the underlying answer while still failing to honor the demanded format, or it might produce well-formed structure whose content drifts under reformatting. Separating these dimensions is useful because enterprise pipelines often depend as much on parseable, schema-valid output as on the answer itself.

The framing connects to a broader industry conversation about structured generation. Providers have introduced features such as JSON mode, function or tool calling, and constrained decoding to force outputs into predictable shapes, and open-source libraries built around grammar-constrained sampling pursue the same goal. Those mechanisms reduce format failures at generation time, but they do not necessarily address the deeper sensitivity the paper targets, namely that the model's reasoning or chosen answer can shift with the wrapper even when the output shape is guaranteed. FSI is positioned as a diagnostic that sits upstream of these tools, characterizing sensitivity rather than merely suppressing its visible symptoms.

The findings, as summarized, highlight reliability gaps in current benchmarking practices. If leaderboard rankings can be nudged by prompt formatting alone, then small differences between models may reflect evaluation artifacts rather than genuine capability. This echoes earlier research showing that few-shot example ordering, delimiter choices, and prompt phrasing can substantially move measured accuracy. The contribution here is less the discovery that sensitivity exists and more the attempt to standardize its measurement so that robustness can be reported alongside raw performance.

For enterprises, the practical implication is that a model performing well on a public benchmark may still behave inconsistently once integrated behind an application's specific prompt template. Teams building retrieval-augmented systems, agents, or automated data extraction increasingly rely on stable, schema-conformant responses, and an unstable model can introduce silent failures that are hard to detect without deliberate stress testing. A metric like FSI could, in principle, inform model selection, regression testing, and prompt hardening.

As with any single-paper proposal, several caveats are warranted. The value of the index depends on how representative its chosen perturbation set is, on the models and tasks evaluated, and on whether the community adopts consistent measurement conventions. It is also likely that some sensitivity is unavoidable given how models are trained on heterogeneous formatting. Independent replication and broader benchmarking across model families would help establish whether FSI becomes a durable evaluation standard or one of several competing robustness measures now emerging in the literature.