Gemma 4 12B を Core ML で 128K コンテキストで動かすThis article explains how to run Gemma 4 12B with a 128K context window on…

Running a 12-billion-parameter language model entirely on a personal machine, with a context window large enough to hold a small book, was until recently the preserve of cloud servers fitted with dedicated accelerators. A recent walkthrough published on Zenn describes how to deploy Gemma 4 12B on Apple Silicon using Core ML while extending the usable context to 128,000 tokens. It is a practical illustration of how far on-device inference has come, and why the combination of capable open models and efficient local runtimes matters for developers.

The central appeal is that everything runs locally. There is no round trip to an external API, no per-token billing, and no need to send potentially sensitive input off the device. For applications that process long documents, large codebases, or lengthy multi-turn conversations, a 128K-token window on hardware a developer already owns is a genuinely useful capability, and it removes a dependency on cloud infrastructure that can complicate both cost and privacy.

Several pieces have to fit together to make this work. Gemma is Google's family of open-weight models, released with published weights that can be downloaded, inspected, and converted rather than accessed only through a hosted service. Core ML is Apple's on-device machine learning framework, which can schedule computation across the CPU, GPU, and the Neural Engine depending on the model and the operations involved. Apple Silicon's unified memory architecture is what makes larger models feasible on consumer hardware, because the CPU and GPU share a single pool of high-bandwidth memory rather than copying tensors across a narrow bus.

The article walks through the conversion and deployment steps, which typically begin with taking the model in its original format and exporting it to the Core ML representation. This stage usually involves quantization, reducing the numerical precision of the weights so that a 12B model fits within the memory budget of a laptop or desktop. Lower precision reduces the memory footprint and can improve throughput, though it may introduce some quality trade-off depending on the scheme used. Handling a 128K context adds a further constraint, because the key-value cache that stores attention state grows with sequence length and can consume a large share of memory on its own, so careful management of that cache is important for long-context runs.

Context helps place the work. Core ML is not the only route to local inference on a Mac. Apple's MLX framework has become a popular option for running and fine-tuning language models on Apple Silicon, and community tools such as llama.cpp and Ollama, which rely on the GGUF format and Metal acceleration, are widely used for the same purpose. Each has different trade-offs in setup complexity, performance, and how directly it taps the Neural Engine. Choosing Core ML in particular can be attractive when the goal is tight integration with an Apple platform application, where a model packaged in Core ML sits naturally alongside other system frameworks.

The broader trend is toward on-device AI as a first-class option rather than a fallback. Model vendors have been releasing smaller, more efficient open-weight families, and hardware makers have been increasing the memory and neural-accelerator capacity of consumer chips. Together these shifts make it plausible to run models that a couple of years ago would have required a server. It is worth noting the practical limits: a 12B model, even quantized, still demands a substantial amount of RAM, and 128K-token inference is likely to be slower and more memory-hungry than typical short prompts, so real-world performance will depend heavily on the specific machine and configuration.

For readers who want to reproduce the setup, the value of a guide like this lies in the specifics of the conversion pipeline and the memory tuning required for long context, which are the parts most likely to trip up newcomers. As open models and Apple's tooling continue to evolve, the exact details of any given procedure may change, but the underlying approach of converting an open-weight model to Core ML and running it locally appears increasingly viable for everyday development. Anyone considering it should benchmark on their own hardware, since the balance between context length, speed, and available memory is where the real constraints emerge.