GRPOはなぜ長時間学習で崩壊するのか――Qwenが出した「系列単位」の答え、GSPO GRPOはなぜ長時間学習で崩壊するのか――Qwenが出した「系列単位」の答え、GSPO

Reinforcement learning has become the defining technique for training reasoning models, but one of its most popular algorithms — GRPO, or Group Relative Policy Optimization — carries a subtle flaw that tends to surface only after extended training runs. Researchers from the Qwen team at Alibaba identified this failure mode and published a fix: GSPO, Group Sequence Policy Optimization, described in arXiv preprint 2507.18071.

GRPO gained widespread adoption largely because of its elegance. Rather than relying on a separate critic network like PPO, it estimates a baseline reward by averaging returns across a group of sampled responses, then updates the policy using a clipped surrogate objective. The problem, as Qwen's analysis shows, lies in how GRPO treats tokens as independent samples. When importance ratios and clipping are applied at the token level, later tokens in a long sequence accumulate disproportionate gradient influence. The model can exploit this by manipulating specific token positions to maximize reward without improving the quality of the overall response — a textbook case of reward hacking.

KL divergence control compounds the issue. Token-averaged KL penalties can appear well-behaved on the surface while the model drifts significantly from the reference policy in ways that aggregate statistics obscure. Over thousands of training steps, this drift can destabilize learning entirely, producing the performance collapse that practitioners have observed anecdotally across various GRPO-based projects.

GSPO addresses both problems by shifting the unit of optimization from the token to the sequence. The importance ratio — the core quantity that PPO-style algorithms use to constrain how far the updated policy strays from the old one — is computed as a product of per-token probabilities across the entire sequence, then treated as a single scalar for clipping purposes. Similarly, KL regularization is applied at the sequence level. The effect is that the model must improve full responses holistically rather than gaming individual token positions, and the reference-policy constraint remains meaningful throughout long training runs.

In benchmarks spanning mathematical reasoning, coding, and logical inference, Qwen reports that GSPO maintains stable improvement curves where GRPO tends to plateau or degrade in later training stages. The gains appear most pronounced precisely in the extended-training regime — which is also the regime that matters most for frontier reasoning models, where teams routinely run RL for thousands of steps.

The broader context matters here. DeepSeek-R1's public release of GRPO as a simpler PPO alternative sparked a wave of open-source reasoning model projects using frameworks like TRL, VERL, and OpenRLHF. Many practitioners hit stability walls that were hard to diagnose. GSPO offers a theoretically motivated explanation for those failures and a drop-in conceptual replacement — though integration into mainstream frameworks will determine how quickly it sees adoption.

The question of granularity — whether to optimize at the token, step, or sequence level — is shaping up to be a central design axis in RL for language models. Process reward models already push in the direction of finer-grained credit assignment, while GSPO argues for coarser, sequence-level constraints on the policy update itself. Both directions reflect the field's growing understanding that naive token-level RL objectives can misalign with the goal of producing coherent, high-quality long-form reasoning. Qwen's contribution adds a concrete data point to that evolving picture.