SkillをGitHub Copilotで"育てる" — 実証的プロンプト改善の実践レポートThis article documents applying mizchi's empirical-prompt-tuning methodology in…

Prompt quality is increasingly treated as an engineering problem rather than a matter of luck, and this report puts that idea to the test. It documents an attempt to apply mizchi's "empirical-prompt-tuning" methodology inside GitHub Copilot, using a regex-builder Skill—a component that generates regular expressions—as the subject of a controlled, iterative improvement exercise. The goal is to show that disciplined, test-driven refinement produces more reliable gains than intuition-based tweaking.

The premise is straightforward. Small differences in a prompt, such as the presence or absence of whitespace or the way instructions are structured, can significantly change the quality of a large language model's output. Developers often work by feel, adjusting wording until something "just works," without fully understanding why. The methodology described here aims to replace that guesswork with a repeatable process, treating a prompt-driven Skill as an artifact that can be measured, regression-checked, and improved systematically rather than by intuition alone.

Empirical-prompt-tuning, as proposed by mizchi, a developer known in the Japanese engineering community, borrows heavily from the mindset of test-driven development. Instead of asking whether a prompt feels right, it asks whether the Skill produces correct results against a defined set of cases. Regex generation is a well-suited domain for this because regular expressions have clear, verifiable behavior: a generated pattern either matches the intended strings and rejects the unintended ones, or it does not. That makes it possible to build objective pass/fail checks around the model's output and to score changes in a way that is hard to dispute.

A notable feature of the experiment is that the author deliberately degraded the regex-builder Skill before improving it. By intentionally lowering the prompt's quality, the report establishes a known, weakened starting point, then demonstrates whether test-guided edits can recover and surpass the original behavior. This resembles an ablation study: beginning from a degraded baseline makes the effect of each change easier to observe than starting from an already-tuned prompt, where improvements tend to be marginal and difficult to attribute to any single edit. The iterative loop—run the tests, inspect failures, adjust the prompt, and re-run—mirrors how engineers debug conventional code.

The work is carried out with GitHub Copilot as the execution environment, which situates it within the broader move to make coding assistants extensible. The notion of a "Skill" reflects a wider trend toward packaging model capabilities as modular, reusable units defined largely in natural language and structured instructions. Comparable ideas appear across the industry, including custom instructions, agent-style extensions, and the general practice of composing specialized behaviors on top of a general-purpose model. Framing a regex generator as a discrete Skill makes it a convenient, self-contained target for this kind of measurement.

For readers new to the space, some background helps. Prompt engineering has matured from ad hoc phrasing into a practice that increasingly resembles software development, complete with version control, evaluation suites, and regression testing. The industry term for systematic measurement here is "evals," where prompts or agents are graded against curated datasets. Approaches like the one in this report apply that discipline at a smaller, developer-facing scale, using the LLM's own outputs as the thing under test. The underlying assumption is that model behavior, while non-deterministic, is stable enough that consistent test results can guide meaningful decisions.

The findings should be read with appropriate caution. This is a single practitioner's account applied to one narrow task, so the results are illustrative rather than definitive, and outcomes are likely to vary with the model version, the phrasing of test cases, and the domain. Regular expressions offer unusually clean success criteria; tasks with more subjective or open-ended outputs would be harder to evaluate this way and may require more elaborate scoring. The report also appears to depend on the specific behavior of the Copilot environment at the time of writing, which can shift as underlying models are updated.

Even with those caveats, the exercise offers a practical takeaway. Treating prompts as testable artifacts, and validating changes against concrete cases rather than impressions, gives developers a more defensible way to improve Skills. For teams building on GitHub Copilot or similar assistants, adopting a test-driven loop appears to be a reasonable step toward more predictable and maintainable prompt-based components.