ACRL：訓練-推論エンジン乖離の適応制御でFP8量子化下のRL学習を安定化Huawei's ACRL framework monitors the discrepancy between training…

Reinforcement learning has become one of the primary levers for improving large language models, yet a subtle piece of infrastructure plumbing can quietly undermine it. Huawei researchers have proposed a framework called ACRL that tackles the discrepancy between the engine used to train a model and the engine used to generate its training samples. The problem matters because when these two systems disagree at the numerical level, a run that is meant to be on-policy becomes effectively off-policy, and that mismatch can escalate into full training collapse.

The issue stems from how contemporary RL pipelines for LLMs are assembled. Gradient updates are typically handled by training frameworks such as FSDP or Megatron, while the rollouts—the model's own generated responses that the algorithm learns from—are produced by dedicated high-throughput inference engines like vLLM or SGLang. These two halves use different implementations and, increasingly, different numerical precisions. A common configuration generates rollouts in FP8 for speed while computing gradients in BF16. As a result, the probability distribution the inference engine actually sampled from is not quite the distribution the trainer assumes it sampled from.

That gap is consequential for on-policy algorithms. Methods such as PPO assume the data being learned from came from the current policy, so any divergence between the sampling and training distributions miscalibrates the implicit importance weights in the update. The gradients become biased or excessively noisy, and over many steps that error can compound until the model's performance degrades sharply. FP8 quantization makes the effect worse because its lower precision widens the numerical distance between the inference and training passes.

ACRL's response is to measure the discrepancy as it happens and compensate at fine granularity. According to the description, the framework adaptively monitors the degree of divergence and adjusts per-token gradient weights, reweighting tokens where the inference and training probabilities diverge most. This is broadly in the spirit of importance-