AIの評価を報酬にする強化学習は何をしているのか — GRPOの1ステップを数字で追うThis article walks through a single GRPO optimization step with concrete…

AI Evaluation as Reward

The article explains how an evaluation produced by an AI can become a reward signal for training a language model. This matters because a score is only a number, making it difficult to picture how that number ultimately changes billions of model weights.

In Reinforcement Learning from AI Feedback, or RLAIF, an LLM scores responses instead of a human, and those scores are used for reinforcement learning. The article also notes increased research into Reinforcement Learning with Verifiable Rewards, or RLVR, which uses mechanically verified correctness, as well as reward systems based on rubric scoring.

SFT as Comparison

Before introducing reinforcement learning, the explanation uses supervised fine-tuning, or SFT, as a comparison. SFT training data consists of prompt-and-correct-response pairs. Training maximizes the probability that the model will produce the token sequence in the correct response, or equivalently minimizes a loss representing that objective.

Here, x is the prompt, y* is the correct response, and πθ is the probability that the current model, with parameters θ, assigns to the response. The essential point is that y* enters the loss function directly. Gradient descent can therefore calculate, from the derivative of the loss, a direction that increases the probability of that correct token sequence.

SFT Training Data

This arrangement means SFT requires correct response text. For each task, that response must be written by a person or distilled from a stronger model.

The article also identifies limits in how this setup handles undesirable answers. There is no place in the described loss for information that says a particular response is bad, so negative examples cannot be incorporated in that way. It also characterizes SFT as learning to reproduce someone else’s writing style.

A Single GRPO Step

The intended reader already understands SFT but has no reinforcement-learning experience. Using GRPO as the teaching method, the article follows one optimization step with concrete numbers.

The walkthrough traces how an LLM-as-a-Judge score is processed as a reward signal until it changes the model’s weights. Its purpose is to make the mechanism intuitive at the level of a single training step, rather than leaving “the evaluation score becomes the reward” as an abstract description.