GGUF互換なのに llama.cpp と PPL が一致しなかった──Layer 0まで追って見えた「演算経路」の違いAn investigation into why perplexity scores diverged between llama.cpp and…

An investigation published on Zenn examines a subtle but consequential problem in local large language model deployment: two runtimes that both load the same GGUF file, and are therefore assumed to be interchangeable, can produce different perplexity scores. The finding matters because perplexity is one of the most widely used yardsticks for judging model quality and quantization loss, and developers frequently compare numbers across tools on the assumption that a shared file format implies identical computation. The author traced the discrepancy all the way down to Layer 0 and concluded that differences in the tensor computation path, rather than the model weights themselves, were generating the divergence.

To understand why this is surprising, it helps to review what GGUF actually guarantees. GGUF is the container format used by llama.cpp and its ecosystem, storing model weights, quantization metadata, tokenizer information, and architectural hyperparameters in a single file. It succeeded the earlier GGML and GGJT formats and was designed to make models portable and self-describing. What GGUF standardizes, however, is the data at rest. It does not prescribe the exact sequence of floating-point operations a runtime must perform to turn those weights into logits. Two implementations can both read identical bytes and still differ in how they multiply, accumulate, and normalize them.

Perplexity is the exponentiated average negative log-likelihood the model assigns to a reference text, so it is highly sensitive to small shifts in the output distribution. Because floating-point arithmetic is not associative, the order in which operations are summed, whether intermediate results are held in FP16 or FP32, and how reduction steps are batched can all nudge the final logits. These tiny per-token differences accumulate across a long evaluation corpus, and the exponential nature of the metric can make them visible even when individual token probabilities barely move. In that sense, a small gap in perplexity between two GGUF-compatible runtimes does not necessarily indicate a bug so much as a difference in numerical bookkeeping.

By narrowing the comparison to Layer 0, the earliest transformer block, the author was able to isolate where the two paths began to diverge before errors propagated and compounded through the remaining layers. Examining the first layer is a sensible diagnostic strategy: if the activations already differ there, the cause lies in the low-level operations rather than in some later architectural detail. The reported conclusion is that the computation route, meaning which kernels are dispatched and how operations such as matrix multiplication, dequantization, and normalization are fused or ordered, differs enough between the implementations to seed a measurable gap. This aligns with a broader and well-documented reality that numerical reproducibility across hardware backends and libraries is difficult to achieve exactly.

The context around this issue is worth spelling out because many popular local tools sit on top of, or alongside, llama.cpp. Runtimes and front ends such as Ollama, LM Studio, KoboldCpp, and text-generation-webui commonly use the llama.cpp backend, but others reimplement inference or use different math libraries, and CPU versus GPU execution, or CUDA versus Metal versus Vulkan kernels, can each follow their own path. Frameworks outside the GGUF world, including Hugging Face Transformers, vLLM, and Apple's MLX, apply their own precision policies as well. Details like whether a BLAS library is used, whether operations run in fused form, and whether accumulation happens in higher precision all influence the outcome. None of these choices is inherently wrong, but they mean that a perplexity figure is only strictly comparable within the same runtime, configuration, and hardware.

The practical takeaway is that format compatibility and numerical reproducibility are distinct properties, and conflating them can lead to misleading conclusions when benchmarking quantization schemes or comparing tools. For developers who need reproducible measurements, the investigation implies that pinning the runtime, its version, the compute backend, and precision settings is as important as pinning the model file. Small perplexity gaps between GGUF-compatible implementations appear to be an expected consequence of divergent computation paths rather than evidence that one runtime is loading the model incorrectly, though larger discrepancies would still warrant closer inspection. The broader lesson is that reproducibility in machine learning inference is a whole-stack concern.