Gemma 4 12BをiPhoneで投機デコードする：2.4倍高速化とA19最適化This article details how speculative decoding applied to Gemma 4 12B on Apple's…

Running a 12-billion-parameter language model on a phone was, until recently, more of a benchmarking stunt than a practical proposition. A recent Zenn post argues otherwise, detailing how speculative decoding applied to Gemma 4 12B on Apple's A19 chip delivers up to a 2.4x improvement in inference speed on an iPhone. The result matters because it targets one of the hardest constraints in on-device AI: making a genuinely capable model fast enough to feel usable without relying on a cloud connection.

Speculative decoding is the core technique. Instead of generating text one token at a time with the full model, the approach pairs a small, fast "draft" model with the large "target" model. The draft model proposes several tokens ahead, and the target model verifies those candidates in a single forward pass, accepting the ones that match what it would have produced anyway and discarding the rest. Because the expensive large model runs fewer sequential steps, throughput rises while the output distribution remains identical to standard decoding. The speedup depends heavily on the acceptance rate: when the draft model's guesses align closely with the target, more tokens are confirmed per pass, and the reported 2.4x figure appears to reflect a well-matched draft-target pairing for Gemma 4 12B.

The choice of hardware is central to the story. Apple's A-series chips combine CPU, GPU, and a Neural Engine on a single package with unified memory, which lets the model weights be shared across compute units without costly copies. On mobile devices, the binding constraint is usually memory bandwidth rather than raw compute, and this is precisely where speculative decoding helps. Each forward pass through a large model must stream its weights from memory, so reducing the number of passes reduces the bandwidth pressure that typically throttles token generation. The article's focus on A19-specific optimization suggests tuning that takes advantage of the chip's memory subsystem and accelerator layout rather than a generic port.

Some prerequisites make the numbers plausible. A 12B model in full 16-bit precision would demand roughly 24 GB of memory, which is beyond current iPhone budgets, so quantization is almost certainly involved. Four-bit or similar low-bit weight formats shrink the footprint to a few gigabytes, at some cost to accuracy, and are now standard practice for on-device deployment. The draft model must also be small enough to add little overhead while still guessing well, so it is likely a distilled or lower-parameter sibling from the same family, which tends to improve acceptance rates because the two models share vocabulary and training lineage.

It helps to situate this within the broader tooling landscape. Gemma is Google's family of open-weight models derived from the same research as Gemma's larger Gemini siblings, and open weights are what make this kind of independent optimization possible in the first place. On Apple hardware, developers commonly reach for MLX, Apple's array framework built for Apple Silicon, or Core ML for accelerator access, while the widely used llama.cpp and its Metal backend have popularized both quantization and speculative decoding across platforms. Speculative decoding itself is not new; it was introduced in research from Google and DeepMind and has since been adopted in server-side inference stacks such as vLLM and TensorRT-LLM. Applying it under the tight memory and thermal limits of a phone is the harder and more novel part.

The industry context explains the interest. Apple, Google, Qualcomm, and others have been pushing inference toward the edge to improve latency, preserve privacy by keeping data on the device, and reduce the recurring cost of server-side generation. Apple Intelligence and Google's on-device Gemini Nano reflect the same trajectory, though those systems generally use far smaller models than 12B. Demonstrating a mid-sized open model at interactive speeds on a handset, if the results hold up across sustained workloads, would narrow the gap between what runs locally and what still requires a data center.

A few caveats are worth keeping in mind. Benchmark speedups from a single blog post can vary with prompt length, batch settings, temperature, and thermal throttling during longer sessions, and a 2.4x figure likely represents favorable conditions rather than a guaranteed floor. Real-world quality also hinges on the quantization scheme chosen. Even so, the write-up is a useful, concrete look at how established techniques combine to make large-model inference practical on consumer hardware, and it offers a reproducible direction for developers experimenting with local LLMs.