Voicebox に Jetson 8GB の Bonsai 27B を繋いだ話 — OpenAI互換APIをRustで全部書いた理由The author ran Bonsai 27B on a Jetson with 8 GB RAM and built a full…

Running a 27-billion-parameter language model on a compact edge device is an ambitious goal, and a recent post on the Japanese developer platform Zenn documents one attempt to make it work end to end. The author describes getting Bonsai 27B to run on an NVIDIA Jetson board equipped with just 8 GB of memory, then writing a complete OpenAI-compatible API server in Rust from scratch so that Voicebox, a voice-oriented application, could send requests to the local model as though it were a hosted cloud service. The result matters because it shows how far local, on-device inference has come, and how much integration glue is still required to connect the pieces.

The hardware constraint is the central challenge. NVIDIA's Jetson modules use a unified memory architecture, meaning the CPU and GPU share the same pool of RAM, so an 8 GB board must hold the operating system, the runtime, and the model weights all at once. A 27B model stored at full precision would need far more memory than that, so fitting it into 8 GB almost certainly depends on aggressive quantization. Depending on how Bonsai is built, that likely means low-bit weights — 4-bit or lower, and possibly ternary-style representations — that trade some accuracy for a dramatically smaller footprint. The write-up frames this squeeze as the main reason the project required careful engineering rather than an off-the-shelf tool.

The choice to expose the model through an OpenAI-compatible API is a pragmatic one. The request and response format popularized by OpenAI's chat completions endpoint has become a de facto standard, and a large number of client applications, including voice frontends, now speak it by default. By implementing that same interface locally, the author could point Voicebox at the Jetson without modifying the client, swapping a remote endpoint for a local one. This pattern is common across the ecosystem: tools such as Ollama, llama.cpp's server, vLLM, LM Studio, and Hugging Face's text-generation-inference all offer OpenAI-compatible endpoints for similar reasons.

Given that those tools already exist, the decision to write the server in Rust rather than reuse them is the part the author spends time justifying. The stated reasoning centers on control and efficiency. Rust compiles to a single native binary with no interpreter or heavy runtime, which is attractive on a memory-limited device where every megabyte counts. Its memory-safety guarantees and predictable performance also suit a long-running server, and handling streaming token output, request routing, and the exact qu