ローカル8Bモデルのツール呼び出し成功率を84%に引き上げる「Forge」、信頼性は配信層の設計で決まるA comparison of local 8B model deployments shows tool-call success rates…

Running an agent on a local large language model often comes down to a frustrating question: why does the same model behave so differently from one setup to the next? A recent comparison published on Qiita puts a striking number on that problem, reporting that identical 8-billion-parameter weights running on the same machine produced tool-call success rates ranging from roughly 7 percent to more than 83 percent, depending entirely on the software layer responsible for serving the model.

The tool built to demonstrate the higher end of that range is called Forge, which the author positions as an inference-serving layer rather than a new model. The headline figure, a tool-call success rate lifted to around 84 percent, is attributed not to a smarter or larger model but to how the serving side formats prompts, constrains output, and parses the results.

For anyone who has built an agent on a local LLM, the failure mode will be familiar. The model is asked to call a tool and is expected to emit a structured payload, usually JSON, describing the function name and its arguments. When that payload is malformed—an unescaped quote, a trailing comma, a hallucinated field, or prose wrapped around the JSON—the agent's execution loop breaks. The model may be perfectly capable of choosing the right action, but if the surrounding infrastructure cannot reliably extract a valid call, the agent fails anyway.

The serving layer influences this in several concrete ways. Chat templates determine how the system prompt, tool definitions, and conversation history are assembled into the exact token sequence the model was trained to expect; a mismatch here can quietly degrade behavior. Constrained decoding, sometimes implemented through grammars or JSON schemas, can force the model to produce only tokens that keep the output syntactically valid. Tool-call parsing then has to map the raw text back into a structured object, handling the many small formatting variations that models produce in practice. Differences in any of these stages appear to explain much of the gap the comparison describes.

This framing aligns with a broader trend in the local LLM ecosystem, where the serving stack has become as important as the model checkpoint. Projects such as llama.cpp, Ollama, vLLM, and Hugging Face's Text Generation Inference each make different choices about templating, sampling, and structured output support. Function calling and JSON mode, popularized by hosted APIs, have pushed open-source servers to add features like guided decoding and schema enforcement, and libraries such as Outlines and XGrammar exist specifically to constrain generation. The Qiita comparison can be read as a practical illustration of why those features matter, especially for smaller models.

The emphasis on 8B models is notable because that size class is a common target for local deployment. Models in this range fit on a single consumer or prosumer GPU and run at usable speeds, making them attractive for privacy-sensitive or cost-sensitive agent workloads. But smaller models are also less forgiving; they are more prone to formatting slips than the largest frontier models, which means the serving layer's guardrails carry proportionally more weight. The claim that reliability hinges on infrastructure design rather than model capability is likely most true in exactly this regime.

Some caution is warranted in reading the numbers. The 7-to-83 percent spread comes from a single author's benchmark, and success rates for tool calling depend heavily on the specific tasks, prompts, and tools tested, as well as on how a success is defined. A different workload could narrow or widen the gap, and the figures should be treated as illustrative rather than definitive. The underlying point, however, is consistent with widely reported experience: for local agents, mundane engineering details in the delivery path often determine whether a capable model is usable in practice.

For practitioners, the takeaway is a reminder to profile the serving layer before concluding that a model is inadequate. Checking whether the chat template matches the model, enabling constrained or schema-guided decoding, and hardening the tool-call parser can matter more than swapping in a larger checkpoint. Whether Forge specifically is the right tool will depend on a given stack, but the comparison it anchors makes a useful argument about where reliability actually comes from. As agent frameworks increasingly assume dependable structured output, the quality of the layer that delivers the model may prove to be the difference between a demo and a system that holds up under real use.