量子化フォーマットを比較する：GGUF・AWQ・GPTQをvLLMで実測するThis article benchmarks GGUF, AWQ, and GPTQ quantization formats on vLLM,…

Quantization has become a practical necessity for anyone running large language models on local or budget-constrained hardware, and the choice of quantization format directly shapes inference speed, memory footprint, and output quality. This article measures three widely used formats — GGUF, AWQ, and GPTQ — on vLLM, a high-throughput inference engine, to give practitioners concrete data rather than folklore when deciding which format to deploy.

To set context, quantization reduces the numerical precision of a model's weights, typically from 16-bit floating point down to 4-bit or 8-bit integers. Fewer bits per weight means a smaller model that fits in less GPU memory and can often be read from memory faster, which is frequently the bottleneck during token generation. The trade-off is potential accuracy loss, since compressing weights discards information. The three formats compared here take different approaches to managing that trade-off, and they emerged from different corners of the ecosystem, which explains some of their behavioral differences.

GGUF is the format associated with llama.cpp and its broad tooling ecosystem, including Ollama and LM Studio. It was designed primarily for CPU and mixed CPU/GPU inference and supports a wide range of quantization levels, from aggressive 2-bit variants to near-lossless 8-bit schemes. GPTQ is a post-training quantization method that uses second-order information to minimize error introduced during weight rounding, and it has long been popular for GPU inference. AWQ, or Activation-aware Weight Quantization, takes the position that not all weights matter equally; it identifies and preserves the salient weights that most influence activations, aiming to retain accuracy at low bit widths. Both AWQ and GPTQ are well suited to GPU execution, which makes them natural candidates for a vLLM benchmark.

vLLM itself is relevant to the comparison because it is optimized for serving throughput through techniques such as PagedAttention, which manages the key-value cache more efficiently, and continuous batching, which keeps the GPU busy by dynamically scheduling requests. Because vLLM was originally built around GPU-native execution, formats designed for GPUs may integrate more naturally than GGUF, which is rooted in the llama.cpp world. Support for GGUF in vLLM has been evolving, and readers should note that results can shift with engine versions, so any single benchmark is best read as a snapshot rather than a permanent verdict.

The article evaluates the formats along three axes: inference speed, usually expressed as tokens per second or latency; memory usage, meaning how much GPU VRAM the loaded model and its runtime state consume; and accuracy, assessed through task performance or perplexity. These three metrics tend to trade off against one another, and the practical value of the piece lies in showing where each format lands. In general terms, formats that preserve more information tend to score better on accuracy but may use more memory, while more aggressive quantization frees memory at some risk to quality. The measured differences in speed often depend as much on how well the format is integrated into the inference kernel as on the bit width itself.

Several caveats are worth keeping in mind. Benchmark outcomes are sensitive to the specific model, the hardware, batch size, sequence length, and the exact software versions involved, so the numbers reported here are likely representative of the tested configuration rather than universally applicable. Accuracy measured on one benchmark task may not transfer to another, and a format that appears fastest at small batch sizes may behave differently under heavy concurrent load. Practitioners are generally advised to reproduce measurements on their own workloads before committing to a format.

For readers building a local LLM stack, the broader takeaway is that format selection is a deployment decision shaped by constraints rather than a search for a single best option. If GPU memory is the binding limit, an aggressively quantized format may be the only way to fit a larger model. If serving throughput on a GPU is the priority, an engine-native format such as AWQ or GPTQ on vLLM appears the more natural fit, whereas GGUF remains compelling for CPU-centric or heterogeneous setups through tools like Ollama and llama.cpp. Measuring on your target hardware, as this article does, remains the most reliable way to choose.