LLMの監督モデルはどこまで小さくできるか。9B・35B・自己検査で実測したら三様の結果になったAn empirical study comparing 9B, 35B, and self-check supervision strategies for…

A growing question for teams running language models on their own hardware is how much compute a verification layer really needs. This experiment report tackles that directly by pitting three supervision setups against one another on a local LLM stack: a compact 9B model acting as supervisor, a larger 35B model in the same role, and a self-check configuration in which the primary model reviews its own output. The finding that the three approaches produced markedly different verification accuracy matters because supervision is increasingly treated as a mandatory safety and quality step, and every extra parameter in that step costs memory, latency, and money.

The core idea behind a supervisor model is straightforward. Rather than trusting a single generation, a second pass evaluates whether an answer is correct, complete, or compliant with instructions before it reaches the user. This pattern appears under several names in the literature, including verifier models, critic models, and the widely used "LLM-as-a-judge" approach. It is closely related to the reward models used in reinforcement learning from human feedback, where a separate model scores candidate outputs. The practical appeal for local deployments is obvious: if a small model can reliably catch the mistakes of a larger one, operators can run oversight cheaply and keep the whole pipeline on-premises without calling an external API.

According to the report, the three configurations did not simply line up in a neat ranking where bigger is better. The 9B supervisor, the 35B supervisor, and the self-check mode each behaved differently enough that the author describes the outcome as three distinct patterns rather than a single trend. That nuance is the most useful takeaway. It suggests that the relationship between supervisor size and verification quality is not linear, and that the choice of strategy interacts with the type of errors being checked. A verifier that is strong at spotting factual contradictions may be weak at judging reasoning steps, and vice versa.

The self-check result is worth isolating because it reflects a known limitation. When a model evaluates its own output, it tends to share the same blind spots that produced the original error, so it can rate a flawed answer as acceptable. Prior research on self-critique and self-consistency has shown mixed results for exactly this reason: a model can sometimes improve by re-examining its work, but it cannot easily detect mistakes that stem from gaps in its own knowledge. Where self-check appears to help most is in catching surface-level slips, such as formatting problems or obvious logical inconsistencies, rather than deep factual errors. The report's empirical framing helps ground these general observations in measured numbers on a specific local setup.

There is also a broader research thread that gives this experiment context. OpenAI's work on weak-to-strong generalization explored whether a weaker model can meaningfully supervise a stronger one, a question that becomes urgent as capable models outpace the humans and smaller models meant to oversee them. Anthropic's Constitutional AI similarly leans on models critiquing and revising outputs against a set of principles. The 9B-versus-35B comparison here is a small, practical instance of the same underlying tension: how far can supervision capacity shrink before it stops being trustworthy. The answer likely depends on the task, the difficulty of the errors, and how the verification prompt is structured.

For practitioners, the implications are concrete. Anyone assembling a local pipeline with tools such as Ollama, llama.cpp, or vLLM must budget VRAM for both the generator and the supervisor, and a 35B verifier roughly quadruples that overhead compared with a 9B one. If a smaller supervisor delivers acceptable accuracy for a given workload, the savings compound across every request. The report stops short of declaring a universal minimum size, and that caution is appropriate, because verification accuracy is heavily task-dependent and a single benchmark cannot generalize to all use cases.

The main value of this write-up is methodological. It offers a reproducible way to ask the minimization question rather than assuming an answer, and it reminds readers that supervisor selection deserves its own evaluation instead of being inherited from the primary model. Teams considering how small their oversight model can go would be well served by running a similar three-way comparison on their own data before committing to a configuration.