Kimi K3と、ペリカンベンチマークから今も学べることKimi K3, and what we can still learn from the pelican benchmark

Simon Willison has put Moonshot AI's newly released Kimi K3 through his signature informal test — asking a large language model to produce an SVG illustration of a pelican riding a bicycle — and used the results to argue that deliberately simple creative prompts remain a surprisingly durable way to separate strong models from weaker ones. The exercise matters because, as standardized leaderboards saturate and concerns about training-data contamination grow, lightweight and idiosyncratic probes are becoming a practical complement for anyone trying to gauge what a model can actually do.

The pelican benchmark works for a specific reason. An SVG is defined entirely in text, coordinates, and path data, so a model must "draw" without seeing what it is producing, translating a mental image into vector shapes and transforms in a single pass. A pelican and a bicycle are each awkward to render on their own, and combining them — perching a large, oddly proportioned bird on a two-wheeled frame — forces the model to reason about spatial relationships, relative scale, and overall composition. Because the request is whimsical and unlikely to appear verbatim in any training set, it resists the memorization that can inflate scores on more conventional evaluations.

Willison reports that Kimi K3 appears to handle the task more coherently than earlier Kimi releases, though he frames the assessment as subjective rather than quantitative. The value of the test, in his telling, is less about crowning a winner than about revealing failure modes: whether wheels are recognizably circular, whether the bird sits in a plausible position relative to the frame, and whether the model maintains internal consistency across dozens of coordinate points. These are qualities that are easy to see at a glance but hard to capture with a numeric score, which is part of why the benchmark has become a recurring fixture across his model reviews.

Kimi K3 is the latest entry in Moonshot AI's model family. The Beijing-based startup has built a reputation around long-context capabilities and, with earlier releases such as Kimi K2, around publishing open weights that developers can run and fine-tune themselves. The article's placement in a local-LLM context suggests K3 is likely intended to be accessible outside a hosted API, positioning it against other openly available systems from Chinese and Western labs. Readers evaluating the model should treat specific capability claims cautiously until independent testing accumulates, since a single creative prompt cannot substitute for broader measurement.

That caveat points to the larger theme of the piece. Formal benchmarks such as MMLU, GPQA, and various coding suites remain important, but they are increasingly vulnerable to two problems: models may have seen the questions during training, and headline numbers cluster so tightly at the top that they no longer distinguish between leading systems. Community-driven approaches like the LMArena preference rankings try to address this with human judgment at scale, while individual reviewers lean on personal, hard-to-game tasks. The pelican test belongs to this second category — a "vibes-based" check that is reproducible enough to compare models over time yet strange enough to stay off the beaten path of optimization.

There is also a practical toolchain behind these evaluations. Willison maintains an open-source command-line utility, llm, that lets users send the same prompt to many different providers and locally hosted models, which makes running an identical test across a dozen systems straightforward. That infrastructure is part of why a one-off joke has evolved into a semi-standard reference point cited by other developers and even acknowledged by some model builders.

The broader takeaway is that model evaluation is becoming a portfolio activity rather than a single scoreboard. As new releases like Kimi K3 arrive at a rapid pace, quick creative probes offer an immediate, intuitive read on progress, while structured benchmarks and real-world usage fill in the rest. The pelican riding a bicycle endures not because it measures anything economically important, but because it remains difficult, memorable, and resistant to gaming — a reminder that useful evaluation sometimes comes from asking a model to do something no one would have thought to train it for.