SageMaker HyperPodでのLLM推論における分離型プリフィルとデコードDisaggregated prefill and decode for LLM inference on SageMaker HyperPod

Amazon Web Services has published technical guidance on running disaggregated large language model (LLM) inference on SageMaker HyperPod, a technique that splits the prefill and decode phases of text generation across separate compute pools. The topic matters because it targets one of the most persistent tensions in production LLM serving: the difficulty of delivering low latency and high throughput at the same time on shared hardware. By decoupling the two phases, operators can scale, tune, and provision each independently.

To understand the approach, it helps to recall that LLM inference proceeds in two stages with markedly different performance characteristics. The prefill phase processes the entire input prompt in parallel, building the key-value (KV) cache and producing the first output token. Because it performs large matrix-matrix operations across all prompt tokens at once, prefill is typically compute-bound and dominates time to first token (TTFT). The decode phase then generates output tokens autoregressively, one at a time, reusing the cached KV state. This stage involves smaller matrix-vector operations and is generally memory-bandwidth-bound, driving inter-token latency (ITL), sometimes called time per output token.

When both phases run together on the same GPUs, a pattern often referred to as colocated serving, they contend for the same resources. A long prefill for one request can stall the decode steps of other in-flight requests, producing latency spikes and forcing awkward tradeoffs in batching strategy. Increasing batch size to raise throughput tends to hurt latency, while optimizing for latency tends to leave hardware underutilized. Disaggregation is intended to break this coupling.

Disaggregated serving, sometimes called prefill-decode separation, assigns each phase to its own group of workers. Prefill workers handle prompt ingestion and first-token generation, while decode workers focus on streaming subsequent tokens. The KV cache computed during prefill is transferred to the decode workers, usually over a high-speed interconnect, so generation can continue without recomputation. Because the two worker pools are independent, each can be scaled to match its own demand, batched according to its own optimal policy, and even placed on different instance types suited to its bottleneck. This makes it easier to meet distinct service level objectives for TTFT and ITL, and to improve overall goodput, the useful throughput that actually satisfies latency targets.

SageMaker HyperPod provides the underlying infrastructure that makes this practical at scale. HyperPod offers resilient, persistent clusters with automated health checks and node replacement, along with orchestration options through Slurm or Amazon EKS for Kubernetes-based deployments. Those features are well suited to a disaggregated topology, where a serving deployment spans multiple heterogeneous worker groups, a router or scheduler, and a mechanism for moving KV cache between nodes. HyperPod's task governance and observability tooling can help track utilization and reliability across the separate pools.

The technique builds on a broader ecosystem of inference software. Open-source engines such as vLLM, SGLang, and NVIDIA TensorRT-LLM have added support for disaggregated execution, and higher-level frameworks including NVIDIA Dynamo and the community project llm-d aim to coordinate prefill and decode workers, request routing, and KV cache handling in Kubernetes environments. Efficient cache transfer libraries, such as NIXL and LMCache, address the network cost of moving KV state, which is one of the main practical considerations. AWS's guidance appears to demonstrate how these components can be assembled on HyperPod rather than introducing an entirely new serving stack.

Several caveats are worth noting. Disaggregation adds architectural complexity and introduces network overhead for KV cache movement, so its benefits are most pronounced at larger scale and with substantial, sustained request volume. Workloads with very short prompts or minimal generation may see limited gains, and the optimal ratio of prefill to decode workers depends heavily on traffic patterns and model size. Tuning that balance is likely to require experimentation and ongoing monitoring.

In the wider context, this work reflects a general industry shift toward treating inference as a distributed systems problem rather than a single-model deployment. As agentic applications and long-context reasoning increase both prompt lengths and generation demands, the economics of serving become more sensitive to hardware efficiency. For teams already invested in AWS, the approach sits alongside fully managed alternatives such as Amazon Bedrock, offering more control over the serving stack in exchange for greater operational responsibility. Organizations weighing it should compare the engineering effort against the potential gains in latency, throughput, and cost per token for their specific traffic.