Skill でトークン消費は減る？GitHub Copilot CLI で計測してみたら「単発計測の罠」にハマった話 An experiment using GitHub Copilot CLI to test whether Skills reduce token consumption fou…

A recent hands-on report published on Qiita examines a practical question that many developers using AI coding assistants are starting to ask: do Copilot Skills actually reduce token consumption, and can that reduction be reliably measured? The author set out to test this using GitHub Copilot CLI, but the more interesting takeaway turned out to be methodological. A single measurement, it appears, can easily mislead, and the piece serves as a cautionary tale about how not to benchmark large language model behavior.

To understand why this matters, it helps to recall what is being measured. Token consumption refers to the number of tokens an interaction sends to and receives from a model, which directly affects both cost and how much of a limited context window is used. Copilot Skills are modular capabilities that can be invoked to handle specific tasks, and the hope is that by delegating certain work to a purpose-built skill rather than stuffing everything into a prompt, the overall token footprint might shrink. If that hypothesis held, teams running Copilot at scale could see meaningful savings on both spend and latency.

The experiment used GitHub Copilot CLI, the command-line interface that lets developers interact with Copilot from the terminal, as the instrument for capturing token counts. On the surface this is a reasonable setup: run a task with a skill, run a comparable task without it, and compare the reported numbers. The author's initial single-run comparison seemed to show a difference. The problem was that when the same test was repeated, the results did not hold steady. The numbers moved around from run to run, sometimes enough to reverse the apparent conclusion. This is the "single-measurement trap" referenced in the title.

The underlying reason is that language model outputs are generally nondeterministic. Unless sampling is fully constrained, the model may produce different responses to identical inputs, and different responses consume different numbers of tokens. Factors such as sampling temperature, prompt construction, how the assistant decides to route a request, and whether any caching or context reuse is in play can all introduce variation. On top of that, the surrounding system may inject additional context, retry steps, or tool calls that are not obvious from a single observed number. Any one of these can make a lone data point unrepresentative of the true average behavior.

The report's central recommendation is therefore straightforward but worth restating: meaningful evaluation requires repeated trials and a statistically grounded approach. Rather than trusting one run, the author suggests collecting many samples for each condition, then comparing distributions rather than individual values. Looking at means, variance, and the spread of results makes it possible to judge whether an observed difference is real or simply noise. Without that discipline, a benchmark can produce a confident-sounding conclusion that fails to reproduce, which is arguably worse than no measurement at all.

This lesson is not unique to Copilot. It echoes a broader concern in the AI tooling space, where informal benchmarks circulate quickly and are often based on small samples. Adjacent tools raise the same challenges. Comparable command-line agents and IDE integrations, along with usage dashboards that report token or request counts, all depend on measurement practices that account for variability. The general principles of experimental design, such as controlling variables, running enough trials to reach statistical confidence, and reporting uncertainty, apply just as they would in any empirical study.

For readers, the practical guidance is to treat token-consumption claims with appropriate skepticism, especially when they rest on a handful of runs. Whether Skills reduce consumption in a given workflow likely depends on the specific tasks, prompts, and configuration involved, and the honest answer may be that it varies. The value of this report is less in delivering a definitive verdict on Skills and more in modeling how to ask the question rigorously. As AI-assisted development tools mature and organizations begin to make budgeting and architecture decisions based on token usage, the ability to measure carefully, and to recognize when a measurement cannot be trusted, becomes an increasingly useful skill in its own right.