AIの評価を報酬にする強化学習は何をしているのか — GRPOの1ステップを数字で追うThis article walks through a single GRPO optimization step with concrete…

Reinforcement learning has become one of the main levers for shaping how large language models behave, from following instructions to solving multi-step math problems, yet the step from a numeric reward to an actual change in the model's weights is rarely explained in concrete terms. A recent blog post on Zenn addresses this gap by following a single optimization step of Group Relative Policy Optimization, or GRPO, with real numbers, aiming to give practitioners an intuitive picture of how AI-generated evaluations function as reward signals.

GRPO sits within the broader family of reinforcement learning from human feedback (RLHF), the approach that helped turn base language models into assistants such as InstructGPT and later chat systems. The dominant algorithm in that lineage has been Proximal Policy Optimization (PPO), which pairs the policy being trained with a separate value network, or critic, that estimates expected future reward. GRPO, introduced by DeepSeek in its DeepSeekMath work and later used prominently in the DeepSeek-R1 reasoning model, removes the critic entirely. Instead of learning a value function, it estimates how good an output is by comparing it against a group of other outputs sampled for the same prompt.

The core mechanic is straightforward once the numbers are laid out. For a given question, the current policy generates a group of candidate answers, and each answer receives a reward, either from a learned reward model or from a rule-based verifier that checks, for instance, whether a final math answer is correct. A worked example makes the effect visible: if four answers to a problem earn rewards of one, zero, one, and zero, the group mean is 0.5. GRPO then normalizes each reward by subtracting that mean and dividing by the standard deviation, producing advantages that are positive for the better-than-average answers and negative for the worse ones. This "group relative" normalization is what replaces the critic in PPO.

Those advantages then feed a clipped objective that closely mirrors PPO. For each token, the algorithm computes the ratio between the probability the updated policy assigns to that token and the probability the old policy assigned to it, multiplies it by the advantage, and