NumLeak: 公開数値ベンチマークが基盤モデルの潜在ラベルになる問題 NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

One of the most persistent challenges in evaluating foundation models is distinguishing genuine reasoning from sophisticated pattern recall. A new arXiv paper, NumLeak, takes a focused look at this problem through the lens of numeric benchmarks — and the findings carry broad implications for how the field measures model capability.

The central claim is straightforward but consequential. Numeric data — economic statistics, scientific measurements, financial indicators — saturates pretraining corpora. When benchmarks use such numbers as ground-truth answers, a model may score well not by reasoning about the underlying phenomenon but simply by reproducing values it encountered during training. The authors call this latent label leakage: the answer is, in effect, already embedded in the model's weights before evaluation begins.

The methodological contribution that distinguishes NumLeak is date-conditioned evaluation. Many numeric quantities change over time. By constructing benchmarks that ask for values tied to specific dates — and by varying those dates relative to each model's pretraining cutoff — the authors argue it becomes possible to separate memorization from out-of-sample generalization. A model that nails a 2023 figure but stumbles on a 2024 one is, plausibly, replaying training data rather than reasoning about the world.

This work lands in a well-established debate. Benchmark contamination has been a recognized threat to evaluation validity for years. When OpenAI released GPT-4, critics quickly questioned how much of its strong benchmark performance reflected memorized test sets. Projects like EleutherAI's lm-evaluation-harness and Hugging Face's Open LLM Leaderboard have attempted to build contamination-detection mechanisms, but the problem remains unsolved at scale. NumLeak narrows the scope to numeric domains, where leakage may be especially insidious precisely because numeric answers look objective and verifiable.

The practical stakes are real. Organizations deploying foundation models for financial analysis, scientific data interpretation, or any task requiring numeric precision face a risk of overestimating those models' out-of-distribution capabilities. A model that appears to "know" macroeconomic indicators may simply have memorized the training-era values, performing poorly when those values change. This argues strongly for in-house evaluation on proprietary or time-shifted data before production deployment.

It is worth noting that NumLeak is currently a preprint and has not yet undergone formal peer review. The methodology's robustness — particularly the assumptions underlying how pretraining data composition is estimated — will require scrutiny. Pinning down exactly what numeric data entered a given model's training set is notoriously difficult given the opacity of most foundation model training pipelines.

Still, the conceptual framing is timely. As numeric reasoning benchmarks grow in prominence — driven by interest in models that can assist with quantitative analysis, coding, and scientific work — the integrity of those benchmarks matters more than ever. NumLeak's approach of exploiting temporal variation to expose memorization offers a practical diagnostic that evaluation practitioners could adapt, even if the full theoretical picture remains to be worked out. For anyone responsible for assessing foundation model performance on numeric tasks, this paper is worth close attention.