ローカルLLM動作比較: gpt-oss vs DiffusionGemma vs Qwen3.5—tok/s は仕事の速さではない A three-way local LLM shootout of gpt-oss, DiffusionGemma, and Qwen3.5 finds that tok/s is…

Local large language models have become practical enough that individual developers now routinely run them on consumer hardware, and a common question follows: which model is fastest? A recent Zenn blog post pushes back on that framing by comparing three locally deployable models—gpt-oss, DiffusionGemma, and Qwen3.5—and arguing that tokens per second (tok/s), the headline metric in most benchmarks, is a poor proxy for how quickly a model actually finishes useful work.

The premise is straightforward. Tok/s measures raw generation throughput: how many tokens a model can emit each second once it starts producing output. That number is easy to capture and easy to compare, which is exactly why it dominates leaderboards and marketing. But the author notes that throughput says little about whether the resulting text is correct, complete, or usable without revision. A model that streams 80 tokens per second yet produces code that fails to compile can be slower, in wall-clock terms, than one that emits 30 tokens per second and gets the answer right the first time.

The three models also illustrate why the comparison is not apples-to-apples. gpt-oss belongs to the family of open-weight reasoning-oriented models, which tend to spend many "thinking" tokens before delivering a final answer. Those intermediate tokens count toward tok/s and total generation time even though the user never reads them directly, so a high throughput figure can coexist with a long time-to-answer. Qwen3.5, from the widely used Qwen lineage, represents a conventional autoregressive decoder that generates one token at a time. DiffusionGemma is the most unusual of the three: it is described as a diffusion-based text model that refines an entire sequence in parallel across denoising steps rather than left-to-right. For diffusion models the very notion of tokens per second is ambiguous, because output does not arrive in the same sequential stream, which further undermines cross-architecture comparison on that single axis.

The post suggests several metrics that track real productivity more closely than raw tok/s. Time to first token matters for interactive use, where perceived responsiveness shapes the experience. Total time to a correct, accepted answer—sometimes framed as "goodput" rather than throughput—captures the cost of retries and corrections. And task-level accuracy, measured on the kind of prompts a user actually issues, determines how often fast output has to be discarded. When reasoning models are involved, counting only the final answer tokens, or measuring end-to-end latency, can produce a very different ranking than counting every generated token.

Some context helps explain why these distinctions surface now. Running models locally usually means going through inference stacks such as llama.cpp, Ollama, LM Studio, or vLLM, and applying quantization—reducing weight precision to formats like 4-bit GGUF—to fit models into limited VRAM. Quantization changes both speed and quality, so the same model can post very different tok/s and accuracy figures depending on how it is packaged. Hardware, context length, and batch size add further variance. All of this means a single benchmark number rarely transfers cleanly from one setup to another.

The broader industry appears to be moving in the direction the article describes. Diffusion-based language models, including research systems and commercial efforts that emphasize parallel generation, are being promoted partly on speed, which makes fair comparison against autoregressive models a live issue. Meanwhile, the spread of open-weight releases—OpenAI's gpt-oss, Alibaba's Qwen, and Google's Gemma among them—has given local users genuine choice and, with it, the need for evaluation methods that reflect their own workloads rather than generic leaderboards.

The article stops short of crowning a winner, which is consistent with its argument: the "best" local model likely depends on the task, the tolerance for errors, and whether latency or throughput dominates the user's workflow. Its practical recommendation is to benchmark against representative prompts and to weigh output quality alongside speed, since a model that is fast but frequently wrong is likely to cost more total time than its tok/s figure suggests. For anyone selecting a local LLM, the piece is a useful reminder that the most quotable metric is not necessarily the most decision-relevant one.