ローカルLLMにISMSの適合状況評価を支援させる ― 第3回 評価者の判断を「部品」に作り込む話This third installment explains why passing raw ISMS clause text to a local LLM…

Compliance teams increasingly want to lean on large language models to speed up tedious audit work, but doing so with information security material raises an obvious concern: the documents involved are often confidential. The third installment of a Japanese blog series on Qiita tackles exactly this tension, describing an experiment in which a local LLM running on an ordinary business PC is asked to help assess conformity with an information security management system (ISMS), specifically ISO/IEC 27001, without any data leaving the machine.

The core argument of the piece is straightforward and, for anyone who has attempted it, familiar: handing the raw text of a standard's clauses to an LLM does not produce trustworthy pass or fail verdicts. The author identifies two reasons. First, a single clause frequently bundles several distinct requirements into one sentence, so a model asked for a single judgment has to collapse multiple conditions into one answer. Second, the threshold for compliance—how much needs to be in place to count as passing—cannot be derived by reading the clause alone. That determination lives in the experience and interpretation of a human auditor.

The proposed remedy is to stop treating the clause text as the unit of work and instead encode the auditor's judgment into structured "components." Rather than asking the model to interpret a requirement from scratch, the approach breaks each clause down into its constituent requirements and attaches explicit criteria describing what evidence or state of affairs would satisfy each one. Those criteria—the parts a seasoned assessor would normally supply mentally—become reusable building blocks that the model can apply consistently across evaluations. In effect, the human does the interpretive work once, captures it in a structured form, and the local LLM handles the more mechanical task of matching evidence against predefined conditions.

This framing places the series squarely within prompt engineering practice, one of the tags attached to the post. The technique resembles a broader move away from open-ended prompting toward decomposition: splitting a complex task into smaller, well-defined subtasks that a model can handle more reliably. It also echoes the reasoning behind retrieval-augmented and rubric-based approaches, where the model is given not just a question but a scaffold that constrains how it should answer.

The choice of a local model is central rather than incidental. Running the model on a business PC keeps sensitive ISMS documentation—internal policies, control implementations, audit evidence—inside the organization, which addresses the confidentiality problem that would otherwise discourage the use of cloud-hosted services for this kind of work. That constraint likely shapes the design, since locally runnable models are typically smaller and less capable than the largest hosted systems, making the structuring of prompts into reliable components more important, not less.

Some background helps situate the effort. ISO/IEC 27001 is the international standard for information security management systems, setting out requirements for establishing, maintaining, and continually improving an ISMS. Conformity assessment—checking whether an organization actually meets those requirements—is normally labor-intensive and dependent on expert judgment, which is why automation is attractive and also why it is difficult. The series appears to be documenting a measured, incremental attempt to find where a local LLM can genuinely help and where it cannot, rather than claiming full automation.

As the third part of a continuing record, the post reads less as a finished product and more as an engineering log: it reports what breaks when you take the naive approach and how reframing an auditor's implicit standards as explicit, reusable components improves consistency. The described method does not remove the human from the loop; it relocates the expert's contribution to the design of the components themselves, leaving the model to apply them. For organizations weighing similar tools, the practical takeaway is that reliable results appear to depend far more on how judgment is encoded than on the raw capability of the model being run.