ツール改善がCopilotコードレビューを悪化させた理由と、実際の改善方法Better tools made Copilot code review worse. Here’s how we actually improved it.

GitHub has published an account of how an effort to strengthen Copilot's code review capabilities initially made the feature worse, and how disciplined measurement rather than intuition eventually produced real gains. The story is a useful illustration of a broader lesson emerging across the industry: adding more capability to an AI system does not automatically improve its output, and without rigorous evaluation, teams can ship changes that feel like progress while quietly degrading quality.

Copilot code review is the feature that lets Copilot analyze pull requests and leave inline comments suggesting fixes, flagging potential bugs, or noting style and correctness issues. Like most modern AI features, it is built on large language models orchestrated as an agent, meaning the model can call tools, gather context from a repository, and reason over multiple steps before producing a comment. The intuition that guided the initial improvement effort was straightforward: give the model access to more tools and more context, and it should produce more accurate, more relevant reviews.

According to GitHub's telling, the opposite happened. Expanding the tooling available to the review agent appears to have introduced noise rather than signal. When an agent has more tools to choose from and more context to sift through, it has more opportunities to pursue irrelevant paths, retrieve information that distracts from the actual change under review, or generate comments that are technically plausible but unhelpful. The result was a decline in the precision of reviews, with the model surfacing lower-value or off-target comments that a developer would likely dismiss. This kind of regression is easy to miss because individual outputs can still look reasonable in isolation; the degradation only becomes obvious when measured across a large, representative sample of pull requests.

The turning point, as GitHub describes it, was leaning on systematic evaluation, commonly referred to as building evals. Rather than judging changes by spot checks or subjective impressions, the team assembled datasets of real review scenarios and defined metrics to score whether Copilot's comments were correct, relevant, and actionable. With that measurement infrastructure in place, they could compare versions of the review agent quantitatively, catch regressions that manual inspection had missed, and iterate deliberately. The improvements that followed came from tightening what the agent did rather than expanding it, an approach that reflects a growing consensus that constraint and focus often beat raw capability in agentic systems.

This experience aligns with prerequisite concepts that anyone building on top of LLMs increasingly has to internalize. Context is not free: stuffing a model with more information can dilute its attention and raise the chance of confident but wrong output, a failure mode sometimes discussed under the umbrella of context management or context rot. Tool use adds decision points, and each additional choice can compound error rather than reduce it. These dynamics explain why evaluation-driven development, borrowed in spirit from traditional software testing and machine learning validation, has become central to AI product engineering.

The account also sits within a competitive landscape where automated code review is a crowded and fast-moving space. GitHub's own Copilot competes with tools such as CodeRabbit, Graphite's review features, Qodo, and various offerings layered on models from providers like Anthropic and OpenAI. Many of these products face the same underlying tension between coverage and precision: reviewers that comment too often train developers to ignore them, while reviewers that comment too rarely miss the issues that matter. The credibility of an AI reviewer depends heavily on keeping its false-positive rate low, which makes precision-focused evaluation especially important.

For engineering teams considering or already using AI review tools, the takeaway is less about Copilot specifically and more about method. GitHub's willingness to describe a regression it caused, and the process it used to reverse it, offers a candid look at how AI features are actually refined in production. It suggests that measurable, reproducible evaluation is likely to be a more reliable path to quality than adding features, and that improvements to AI-assisted development are often achieved by doing less, but doing it more carefully and with better instrumentation to prove it worked.