ollama の入力切り捨てをレスポンスだけで検知する — 3回作り直した記録A practical account of detecting silent input truncation in ollama—where…

Running large language models on local hardware has become far more accessible through tools such as ollama, yet that convenience can obscure subtle failure modes that quietly degrade output quality. One of the more insidious is silent input truncation, in which a prompt that exceeds the model's configured context window is cut off without any error or warning. A recent Zenn post documents a practical effort to detect this behavior using nothing but the model's response as evidence, a technique the author says was refined over three separate redesigns.

The issue is worth attention because truncation that goes unannounced can produce answers that look complete but are actually based on partial input. In retrieval-augmented generation, long document summarization, or agent workflows that accumulate history, a silently dropped section can change the meaning of a result without leaving an obvious trace. Because the model still returns fluent text, the failure can persist unnoticed across many runs.

Context windows sit at the heart of the problem. Every model has a maximum number of tokens it can attend to at once, and in ollama this is governed by the num_ctx parameter. Historically, many configurations defaulted to a relatively small window, so prompts assembled from large files or long conversations can exceed the limit more easily than users expect. When that happens, the underlying inference engine trims the input to fit, and the caller typically receives no explicit indication that anything was removed.

An obvious first instinct is to compare token counts. Ollama's API returns fields such as prompt_eval_count, which reports how many prompt tokens were processed, alongside eval_count for generated tokens. In principle, comparing the number of tokens sent against prompt_eval_count could reveal truncation. In practice, this signal is easy to misread: prompt caching can mean the reported count does not correspond to a fresh evaluation of the entire prompt, and tokenization differences make client-side counting an imperfect estimate. These complications appear to be part of why the author pursued a method that leans on the response content itself rather than trusting the numeric metadata alone.

The reported approach evolved through iteration, which is instructive in itself. Early designs likely relied on assumptions that broke under real conditions, such as caching interference or false positives when a model simply chose to answer briefly. Later revisions appear to have moved toward more robust signals, for example planting a recognizable marker near the start of the prompt and checking whether the model can still reproduce or acknowledge it in its reply. If the beginning of the input is truncated, that marker disappears, and the absence becomes a detectable clue. The value of three redesigns is that each exposed edge cases the previous version missed, a familiar arc in debugging nondeterministic systems.

This work connects to a broader set of concerns in local LLM operation. Users of llama.cpp, which underpins ollama, have encountered similar truncation and context-management behavior, and projects like LM Studio, text-generation-webui, and vLLM each handle context limits and overflow differently. Understanding a specific runtime's defaults, and instrumenting prompts to confirm they