「Speculate with Memory」: LLMエージェントの無損失高速化手法Speculate with Memory: Lossless Acceleration for LLM Agents

Researchers have proposed a technique called "Speculate with Memory" that aims to accelerate large language model (LLM) agents by reusing their own past execution traces, without altering the text the model ultimately produces. Described in a paper posted to arXiv, the work targets a growing practical problem: agentic systems that repeatedly call an LLM to plan, reason, and invoke tools tend to be slow and costly, yet much of what they generate closely resembles output they have produced in earlier steps or earlier runs.

The method builds on speculative decoding, an inference-acceleration approach that has become common in production LLM serving. In conventional autoregressive generation, a model emits one token at a time, and each step requires a full forward pass, so latency scales with output length. Speculative decoding breaks this bottleneck by having a cheaper "draft" mechanism guess several tokens ahead, then letting the large target model verify all of them in a single parallel pass. Tokens that match the target model's own distribution are accepted, and the rest are discarded and regenerated. Because the target model still validates every token, the final output is provably identical to what standard decoding would have produced. This equivalence is why such methods are described as lossless, distinguishing them from lossy shortcuts like quantization or pruning that can subtly change results.

The distinguishing idea in this paper is where the draft tokens come from. Instead of relying only on a smaller draft model or on generic heuristics, "Speculate with Memory" treats the agent's prior execution history as a retrieval source for candidate continuations. In repetitive agentic workflows, an LLM often regenerates near-identical spans, such as boilerplate reasoning, structured tool calls, formatted responses, or repeated planning templates. By storing these past outputs and retrieving relevant fragments, the system can propose long candidate sequences that the target model may accept in bulk. When the memory is a good match, many tokens are confirmed per forward pass, and speedups appear to grow with the degree of repetition in the task. When the match is poor, the target model simply rejects the guesses and falls back to normal generation, which preserves correctness at the cost of forfeiting the speedup on those steps.

This positions the work alongside a family of draft-model-free speculative techniques. Prompt lookup decoding and n-gram-based approaches draft tokens by copying spans from the current prompt, which helps in summarization or retrieval-augmented settings where input and output overlap. Retrieval-based methods such as REST pull candidates from an external datastore of text. Other approaches, including Medusa and EAGLE, attach lightweight prediction heads to the base model to generate drafts without a separate model. "Speculate with Memory" is conceptually closest to the retrieval-based line, but it is tailored to the agentic setting, where the most predictive source of future tokens is the agent's own accumulated trajectory rather than a static corpus.

The context that makes this relevant is the rapid expansion of agent frameworks. Patterns such as ReAct interleave reasoning with tool use, and systems built with orchestration libraries frequently loop through similar subtasks across many iterations. These loops multiply inference calls, so per-call latency becomes a dominant factor in overall wall-clock time and serving cost. A method that exploits redundancy across an agent's history is therefore well aligned with how these systems actually behave, and it complements rather than replaces existing serving optimizations like key-value cache reuse, batching, and prefix caching.

Several caveats are worth noting without the full experimental details in hand. The benefits are likely concentrated in workflows with high repetition, and gains may be more modest for open-ended or highly variable tasks. Maintaining and searching a growing memory store introduces its own overhead, and the retrieval step must be fast enough not to erode the savings. The lossless guarantee, however, is a meaningful property: because the approach only changes how tokens are drafted and not how they are verified, adopting it should not degrade an agent's answers. If the reported speedups hold across realistic benchmarks, the technique could offer a low-risk way to reduce latency and compute in production agent deployments.