「自社AIを育てる」前に — ローカルLLM+RAGで検証したら、ファインチューニングは要らなかった A hands-on experiment using bge-m3, LanceDB, and Ollama (gemma) found that a local RAG pip…

The idea of building a company-specific AI that "understands your business" is a compelling pitch, especially to decision-makers eager to differentiate with in-house AI capabilities. But the path to that goal is often misunderstood — fine-tuning a large language model on proprietary data sounds like the obvious approach, yet it may be overkill for the vast majority of real-world use cases.

A post on Zenn puts that assumption to the test. The author constructs a minimal local RAG pipeline — bge-m3 for multilingual embeddings, LanceDB as the vector store, and Ollama running gemma as the inference backend — and evaluates whether it can replicate the results that practitioners typically expect from fine-tuning. The verdict: for canonical enterprise scenarios like document Q&A and domain-specific terminology handling, RAG alone consistently hit practical accuracy thresholds without any model training.

The distinction between what fine-tuning and RAG are actually good at is worth unpacking. Fine-tuning reshapes a model's behavior, tone, and latent knowledge — it excels at teaching a model to respond in a particular style or to internalize a stable body of domain knowledge baked into training data. What it cannot do efficiently is keep up with live, changing information. Every update to an internal wiki or policy document would require a new training run, which is expensive, time-consuming, and operationally fragile. RAG sidesteps this entirely by retrieving the relevant context at inference time, meaning document updates are reflected in answers immediately.

The choice to run everything locally is equally significant. By keeping inference on-premises via Ollama, sensitive corporate documents never leave the organization's infrastructure — a critical consideration for legal, healthcare, and financial verticals. bge-m3, developed by BAAI, supports dense retrieval across CJK languages including Japanese, making it a strong candidate for organizations whose document corpus is not exclusively in English. LanceDB, written in Rust and designed for embedded, serverless deployments, keeps the infrastructure footprint small and the setup overhead low for rapid experimentation.

This experiment lands at a moment when the open-model ecosystem is more capable than ever. As of mid-2026, instruction-tuned models available through Ollama from providers like Google, Mistral, and Meta have closed much of the gap with proprietary APIs for structured enterprise tasks. The practical implication is that the baseline capability a local LLM brings to a RAG pipeline is genuinely high — high enough that the marginal value of fine-tuning is harder to justify unless very specific behavioral constraints are required.

For teams evaluating AI investments, the takeaway is strategic as much as technical: exhaust the RAG option first, measure where it falls short, and only then assess whether fine-tuning addresses the remaining gap. This sequencing avoids costly, premature commitments to training infrastructure and keeps iteration cycles short. As the author frames it, what most organizations call "an AI that grows with the company" is almost always a retrieval problem in disguise — and retrieval, it turns out, is a solved one.