LLMレビューが終わらないのは自分の修正のせいか — 108ラウンドを検証したAn experiment running 108 rounds of dual-LLM review on a operations document…

A developer's experiment putting a long operations manual through 108 rounds of review by two large language models offers a cautionary data point for anyone building automated document-quality pipelines. The setup sounds straightforward, but the result was that the review loop never converged, and the investigation into why points to a subtle failure mode: the edits meant to fix the document may themselves be generating fresh problems for the next round to flag.

The methodology was deliberate. A lengthy operational procedure document was submitted to two LLMs from different providers. Each flag raised by the models was checked and verified rather than accepted at face value, and only genuine problems were corrected. The revised document was then fed back to the same two models, and the cycle repeated. According to the account, this continued for 108 rounds, and across all of them the two models never once simultaneously returned a clean verdict of "no issues." Fixing one item consistently surfaced another; fixing that one surfaced yet another, in a chain that showed no sign of terminating.

That pattern prompted the central question of the experiment. Rather than assuming the models were steadily uncovering a finite backlog of pre-existing defects, the author suspected the opposite: that in the later rounds the reviews were increasingly reacting to defects introduced by the author's own corrections. In other words, each edit could be trading one flaw for another, so the loop was partly chasing its own tail. The stated summary of the findings is that later rounds did appear to flag more issues attributable to prior edits than to original flaws, a tendency described as "edit drift," which is offered as one reason iterative LLM review loops fail to reach a stable end state.

The proposed way to test this is worth noting because it turns a frustrating experience into a measurable claim. If corrections are the culprit, then the proportion of flags caused by recent edits should rise as the review progresses. Tracking whether "fix-induced" flags grow in the back half of the run, relative to flags against untouched original text, gives a signal that can distinguish genuine convergence toward quality from a loop that is merely reshuffling problems. The 108-round length matters here, because a short run could not reveal a trend that only becomes visible over many iterations.

For context, this experiment sits alongside a broader industry interest in using models to critique and improve text, often grouped under labels like LLM-as-a-judge, self-refine, and iterative or reflexive editing. These approaches assume that a model's feedback, applied repeatedly, will monotonically raise quality until little remains to fix. The observation here complicates that assumption. Language models are probabilistic and sensitive to phrasing, so a rewrite can shift wording in ways that a subsequent pass reads as a new inconsistency, ambiguity, or omission, even when the underlying meaning is sound. Using two models from different providers is a reasonable attempt to reduce single-model bias and idiosyncrasy, yet even that cross-check did not produce agreement that the document was finished.

There are practical implications for teams applying such loops to documentation, runbooks, or knowledge bases, including those feeding retrieval-augmented generation systems where source quality directly affects downstream answers. The experiment suggests that "no more flags" may be an unrealistic stopping condition, and that a loop left running indefinitely can degrade as easily as improve. More defensible stopping rules might include capping the number of rounds, requiring that a flag be independently confirmed before any edit is made, or measuring whether new flags concern genuinely material problems rather than stylistic churn.

The findings should be read as a single, informal case study rather than a controlled benchmark. The specific models, the document, and the human judgment applied when verifying each flag all shape the outcome, and results are likely to vary with different material and different model pairings. Even so, the core insight is a useful one: in iterative LLM review, the reviewer and the editor are entangled, and the assumption that more rounds always mean a better document does not necessarily hold.