LLM-as-judgeを疑え — 忠実性スコア3.20の犯人は、答案ではなく採点者だったAn investigation into low faithfulness scores in RAG evaluation revealed the…

Evaluating retrieval-augmented generation (RAG) systems increasingly leans on a technique known as LLM-as-judge, where one language model grades the output of another. A recent write-up on Zenn offers a useful cautionary tale: after a RAG pipeline returned a stubbornly low faithfulness score of 3.20, the author dug into the numbers and concluded that the fault lay with the grader, not the answers it was scoring. The finding is a reminder that an evaluation pipeline can be just as broken as the model it is meant to measure.

Faithfulness is one of the core metrics in RAG evaluation. It asks whether the statements in a generated answer are genuinely supported by the retrieved context, as opposed to being fabricated or drawn from the model's parametric memory. A faithful answer sticks to what the sources say; an unfaithful one hallucinates. Because verifying this by hand across hundreds of examples is slow and costly, many teams hand the job to a second LLM, which reads the question, the retrieved passages, and the candidate answer, then issues a score, often on a scale such as one to five.

That workflow rests on an unstated assumption: that the judge is a trustworthy referee. The Zenn investigation illustrates why that assumption should be tested rather than taken for granted. When the author inspected individual gradings rather than the aggregate average, the low scores did not correspond to genuinely unfaithful answers. Instead, the judge appeared to be making systematic mistakes, penalizing answers that were in fact grounded in the provided context.

Several failure modes are commonly cited for this kind of behaviour, and they map onto what the post describes. Judges can be overly strict or overly lenient depending on how the grading prompt is phrased. They can misread the boundary between what counts as "supported" and what counts as an inference. Smaller or locally hosted models, which are attractive for cost and privacy reasons, may struggle to decompose an answer into individual claims and check each one against the source, leading to noisier scores than a larger frontier model would produce. Formatting quirks, such as an answer that restates the context in different words, can also trip up a judge that is matching too literally.

The practical takeaway is that the evaluation harness needs its own validation. One recommended approach is to calibrate the judge against a set of human-labelled examples, measuring agreement before trusting its verdicts at scale. Inspecting the judge's reasoning, when it is prompted to explain its grade, can reveal whether it is applying the intended criteria or latching onto surface features. Comparing scores across different judge models, or across different prompt templates, helps expose instability. If a change to the grading prompt swings the average score substantially while the underlying answers are unchanged, the metric is likely measuring the judge as much as the system under test.

This episode fits into a broader conversation about the reliability of automated evaluation. Frameworks such as RAGAS, DeepEval, TruLens, and Arize Phoenix have popularised metrics like faithfulness, answer relevancy, and context precision, and many of them implement faithfulness by breaking an answer into atomic claims and verifying each against the context. Research on approaches like G-Eval has explored using chain-of-thought prompting to make LLM graders more consistent. At the same time, studies have documented known biases in LLM judges, including a tendency to prefer longer answers, sensitivity to the order in which candidates are presented, and self-preference, where a model rates its own style of output more highly.

None of this means LLM-as-judge should be abandoned. For many teams it remains the only affordable way to evaluate open-ended generation at volume, and it can correlate reasonably well with human judgement when set up carefully. The lesson from the Zenn post is narrower but important: a suspicious score is a prompt to audit the whole measurement chain, not just the component you were originally trying to improve. Before concluding that a RAG system is unfaithful, it is worth confirming that the judge is faithful too.