AIエージェントの「判定」問題、検査工程はとっくに解いていたThis article argues that the challenge of having AI agents reliably evaluate…

Judgment Engineering

The article argues that reliably evaluating AI-agent outputs is fundamentally similar to a problem manufacturing inspection workflows have addressed for decades. This matters because many ideas missing from AI verifiers may already exist within quality assurance.

The author provisionally calls the field “Judgment Engineering”: the engineering of how evaluation criteria are created, how incorrect judgments are recorded, and how the accuracy of the evaluator itself is verified. Rather than treating judgment as a simple final step, this framing makes the reliability of the judging process an object of measurement and review.

Quality Assurance

Manufacturing inspection establishes criteria for passing or failing, measures whether judgments vary, and later checks whether those judgments were correct. According to the article, inspection processes have performed these tasks for decades within the framework of quality assurance, including through statistical methods.

The parallel applies to AI verifiers and review sub-agents. Their role is not only to produce a judgment, but to do so against criteria whose reliability can itself be examined. The article suggests that manufacturing’s established approach to inspection can inform how those agent roles are designed.

Limit Samples

A limit sample is a physical example showing the boundary between acceptable and unacceptable results in visual inspection. Written criteria can be interpreted differently by different inspectors, so borderline passing and borderline failing samples are prepared and shown to everyone.

The aim is to align judgments through concrete examples rather than words alone. The article compares AI evaluation criteria written mainly as natural-language prompts to asking inspectors to make decisions without limit samples.

Gauge R&R

Gauge Repeatability and Reproducibility, or Gauge R&R, is a form of measurement-system analysis. It statistically measures variation when the same inspector repeatedly measures the same product, known as repeatability, and variation when different inspectors measure it, known as reproducibility.

In the article’s framing, Gauge R&R provides a numerical way to answer whether an evaluation standard can actually be trusted. Applied to Judgment Engineering, the central issue becomes not merely whether a verifier reaches a decision, but whether its judgments remain sufficiently consistent to support confidence in the evaluation process.