Apple Neural Engine で LLM を、出力を変えずに高速化する — Core ML 投機デコードの実装A Core ML bundle running Gemma 4 E2B on Apple Neural Engine gains lossless…

Running large language models directly on Apple's Neural Engine (ANE) is one of the more stubborn problems in on-device inference, and a newly published open-source Core ML bundle offers a concrete attempt at it. The project runs Gemma 4 E2B on the ANE and adds two capabilities aimed at real-world use: lossless speculative decoding that requires no separate draft model, and disk persistence for the key-value (KV) cache. According to the author, the acceleration does not change the model's output by a single byte, which is the central claim worth examining.

Speculative decoding is an established method for speeding up autoregressive text generation. In its typical form, a small, fast draft model guesses several tokens ahead, and the larger target model verifies those guesses in one forward pass, accepting the longest prefix that matches what it would have generated on its own. Because verification preserves the target model's output distribution, the result is identical to ordinary decoding — hence "lossless." What distinguishes this implementation is that it is described as draft-model-free, avoiding the need to load and run a second model. On memory-constrained Apple hardware, that trade-off is significant.

The write-up details several implementation choices. The Core ML package uses a multifunction layout, in which multiple functions share weights inside a single model, and it keeps the KV cache on the host side rather than entirely within the ANE. Verification runs at width 4, meaning a small batch of candidate tokens is checked each round. The author pairs these design notes with measured figures from actual iPhone hardware rather than simulator numbers, and gates the results behind on-device testing.

Persisting the KV cache to disk addresses a practical pain point. The KV cache stores the attention keys and values computed for prior tokens so they need not be recomputed on every step; saving it to disk allows a session to resume without reprocessing an entire prompt. For long system prompts or repeated conversations, this can remove a substantial share of the startup cost, though the benefit depends on how much context is reused.