AIラボは「pelicanmaxxing」をしているのか？Are AI labs pelicanmaxxing?

Simon Willison, the developer and prolific AI commentator, has raised a pointed question about the direction of large language model development: are AI labs "pelicanmaxxing"? The term combines his own informal benchmark — asking a model to generate an SVG image of a pelican riding a bicycle — with the internet suffix "-maxxing," which implies obsessively optimizing for a single outcome. The underlying concern is familiar but increasingly relevant: that model builders may be tuning their systems to score well on visible tests rather than to become genuinely more capable. It matters because the public largely judges new models by exactly these headline results.

The pelican test itself has an unusual origin. Willison began asking models to produce vector graphics of a pelican on a bicycle as a lightweight, memorable way to probe several abilities at once. Generating valid SVG requires a model to reason about coordinates, shapes, and spatial relationships without seeing the rendered output, and drawing a recognizable pelican astride a bicycle — a subject unlikely to appear verbatim in training data — is genuinely hard. Over time the challenge gained a following, with results shared widely and treated as an offhand gauge of each new release.

That popularity is precisely what creates the risk Willison appears to be describing. Once a test becomes well known, labs have an incentive, whether deliberate or incidental, to ensure their models handle it well. This is a specific instance of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. If a model does unusually well on the pelican prompt because its training emphasized that exact scenario, the result no longer says much about the model's broader competence. "Pelicanmaxxing" is a tongue-in-cheek label for this failure mode, but the concern generalizes far beyond one whimsical drawing.

The same dynamic is a long-standing worry for formal benchmarks. Suites such as MMLU, GPQA, HumanEval, and SWE-bench are routinely cited in launch materials, and strong scores can shape media coverage, developer adoption, and investment narratives. Researchers have repeatedly flagged benchmark contamination, where test questions or close variants leak into training data, as well as more subtle overfitting where models are refined against the style of popular evaluations. Crowd-based comparisons like LMArena, which rank models by human preference, were partly a response to these problems, though they introduce their own biases toward answers that simply feel appealing.

Willison's framing is useful because it makes an abstract statistical hazard concrete and slightly absurd. A pelican on a bicycle is obviously not something users need, so if labs were quietly optimizing for it, the gap between benchmark performance and practical value would be laid bare. The broader point is that any widely watched metric can drift from measuring capability to measuring the ability to satisfy that metric. This is especially likely in a competitive market where a handful of leading labs release models on overlapping timelines and compete for the same attention.

There are reasons to be measured about the accusation. It is difficult to prove that any given lab is deliberately gaming a specific test, and much apparent improvement is likely genuine, driven by larger and cleaner datasets, better training techniques, and more capable base models. Optimizing for a memorable challenge is also not automatically dishonest; the danger lies in mistaking a narrow, tuned result for evidence of general progress. Willison seems to be posing a question rather than leveling a firm charge.

The practical takeaway for developers and organizations is to treat any single benchmark, including informal ones, as one signal among many. Private evaluations built from an organization's own tasks are harder to game precisely because labs cannot see them, and they map more directly onto real needs. Techniques such as holding out fresh test sets, rotating prompts, and inspecting outputs qualitatively can help distinguish real capability from surface polish. As the field continues to lean on leaderboards to communicate progress, Willison's coining of "pelicanmaxxing" is a compact reminder that the map is not the territory, and that a model's usefulness is ultimately measured by the work it actually helps people do.