アイドルアクセラレータを最小化：llm-dの協調タイムスライシングによるネイティブRLジョブインターリービングMinimize idle accelerators: Native RL job interleaving with co-operative time-slicing in llm-d

Google Cloud has detailed a new capability in llm-d, the Kubernetes-native distributed inference stack, aimed at one of the more stubborn inefficiencies in large-model development: accelerators sitting idle during reinforcement learning. The feature, described as cooperative time-slicing, allows inference and training jobs to interleave on the same GPUs. According to the project, this can significantly reduce the time that expensive hardware goes unused, which matters as reinforcement learning becomes a central technique for aligning and improving large language models.

The inefficiency the feature targets is structural to how reinforcement learning is applied to modern models. A typical RL loop alternates between two distinct phases. In the generation, or rollout, phase, the current policy model produces sample outputs, a workload dominated by inference. In the subsequent training phase, those samples are scored against a reward signal and used to update the model's weights, a pattern closer to conventional gradient-based training. Because these phases have very different compute and memory profiles, teams have historically had to choose how to place them.

Two approaches have been common, and each has drawbacks. In a disaggregated setup, separate pools of accelerators handle rollout and training. This isolates the workloads but tends to leave one pool idle while the other runs, since the phases execute sequentially in the loop. In a colocated setup, both phases share the same devices, improving utilization but complicating memory management, since inference engines and training frameworks compete for GPU memory and must repeatedly load and unload model weights and optimizer state. The result, in many pipelines, is a bimodal usage pattern in which costly accelerators spend a meaningful fraction of wall-clock time waiting.

Cooperative time-slicing appears to address this by coordinating the two workloads at the software level so they can share accelerators without hard partitioning. Rather than relying purely on hardware mechanisms, the approach lets the inference and training components yield to one another in a scheduled fashion, so that when the training step runs, inference capacity can be released, and vice versa. The "cooperative" framing suggests that the participating jobs are aware of the shared arrangement and hand off resources deliberately, which is generally harder to achieve with opaque, competing processes.

Making this work in practice requires solving several technical problems that the announcement touches on. Weight synchronization is central: after each training update, the newly refined policy must be reflected in the inference engine that generates the next batch of rollouts, and doing this efficiently avoids a costly reload cycle. Memory coordination is equally important, because the KV cache used during inference and the optimizer state used during training cannot both occupy peak capacity simultaneously. By interleaving the phases and managing these transitions natively within the framework, llm-d aims to keep the accelerators productive across a larger share of the loop.

For context, llm-d is an open source effort built around Kubernetes and the vLLM inference engine, developed with contributions from Red Hat, Google, IBM, and others, and positioned as a way to run distributed inference at scale with features such as disaggregated serving and inference-aware routing. Extending it toward reinforcement learning is a natural step, since RL pipelines lean heavily on high-throughput generation, which is precisely what the inference layer is optimized to deliver.

The move also sits alongside broader industry work on sharing scarce accelerators. At the hardware and platform level, NVIDIA offers Multi-Instance GPU for partitioning a device, Multi-Process Service for concurrent kernels, and basic time-slicing, while Kubernetes has been advancing Dynamic Resource Allocation to give schedulers finer control over GPUs. Cooperative time-slicing is likely complementary to these, operating at the workload-coordination layer rather than replacing lower-level mechanisms. In the RL tooling ecosystem, frameworks such as TRL, veRL, NeMo-Aligner, and OpenRLHF have each grappled with the colocated-versus-disaggregated tradeoff, so a Kubernetes-native option that keeps accelerators busy could appeal to teams already standardized on that platform.

As with any single-vendor description of an open source feature, the practical gains will depend on model size, cluster topology, and the specific RL algorithm in use, and independent benchmarks would help clarify the real-world impact. Still, the direction is clear: as reinforcement learning grows more important and accelerator supply remains constrained, reducing idle time by interleaving jobs on shared hardware is an increasingly valuable optimization, and llm-d's approach offers one concrete implementation of that idea.