Voicebox に Jetson 8GB の Bonsai 27B を繋いだ話 — OpenAI互換APIをRustで全部書いた理由The author ran Bonsai 27B on a Jetson with 8 GB RAM and built a full…

Voicebox and Jetson

The author describes running Bonsai 27B on a Jetson Orin Nano with 8 GB of memory and connecting it to the Mac AI voice studio Voicebox. The project matters because meeting the memory and optimization constraints required control that the author could not obtain from existing standard tools.

In the reported test, Voicebox called the model through an OpenAI-compatible server-sent events API over a Tailscale VPN. OpenAI-compatible API and SSE support on the Voicebox side was implemented in a fork. The author submitted the changes upstream as pull requests #947 and #948, both of which were open at the time of writing.

Why the Full Stack

The author considered established local LLM software, including llama.cpp, Ollama, vLLM, and text-generation-inference. Although the article describes these options as fast, stable, and supported by large communities, they did not match the project’s design goals.

The central requirement was fine-grained control over per-layer hybrid inference on an edge device with 8 GB of memory. The desired configuration included assigning DeltaNet layers to the CPU while running Attention layers on the GPU. According to the author, llama.cpp and Ollama could not provide that degree of control.

Pure Rust Inference Engine

The resulting system, ALICE-LLM, is a pure Rust LLM inference engine released under AGPL-3.0+ on GitHub. The author built the stack from scratch, including the GGUF parser, GPU kernels using wgpu, and the OpenAI-compatible SSE streaming server.

This approach gave the author complete control over how memory was owned and managed across the stack, rather than placing the API server on top of an existing inference engine.

Extreme Quantization and Optimization

The implementation combined Bonsai 27B’s Ternary/Q1_0 extreme quantization with work to avoid two Vulkan duplication operations on the Jetson. The article also reports reaching 45% of the M1 Pro CPU’s physical bandwidth limit.

The author presents these constraints as the reason complete memory ownership in Rust was necessary. The broader lesson is narrowly framed: even when mature tools already exist, rebuilding core components can be rational when edge constraints and extreme optimization require controls that those tools do not expose.