LiteRTとNPUで実現するオンデバイスAIの実装 Building real-world on-device AI with LiteRT and NPU

Google has announced a significant expansion of LiteRT (formerly TensorFlow Lite), bringing first-class support for NPUs (Neural Processing Units) from major silicon vendors. The move aims to give developers a unified path to running increasingly large on-device AI workloads with better latency and power efficiency than CPU or GPU execution alone.

According to the announcement, LiteRT now abstracts NPU backends from Qualcomm, MediaTek, Samsung and other SoC partners behind a single API. Historically, tapping into an NPU required vendor-specific SDKs, custom toolchains and substantial porting effort, which made cross-device deployment painful. By owning the delegate and hardware-selection layer, LiteRT lets a model author target multiple chips with the same runtime invocation, reducing the engineering burden of shipping AI features across the fragmented Android device landscape.

The update also addresses practical concerns specific to on-device inference, including efficient execution of quantized models, lower memory footprint, and more predictable latency for interactive use cases. Google is rolling out access through an early access program, suggesting that the company wants to mature the API alongside real production workloads in domains such as imaging, speech, and small generative models before a broader release.

The broader context matters here. NPU performance, measured in TOPS, has been climbing rapidly across recent flagship and even mid-range SoCs, and the rise of compact on-device LLMs — Gemini Nano being the most visible example — has intensified demand for hardware acceleration that goes beyond GPU shaders. Running models locally offers tangible benefits in privacy, offline availability, and per-query cost, all of which are increasingly relevant as AI features move from novelty to default expectation in mobile apps.

LiteRT is not alone in this push. ONNX Runtime, Apple's Core ML with the Neural Engine, Qualcomm's AI Hub, and MediaTek's NeuroPilot all pursue similar goals, and frameworks like MLC LLM and llama.cpp have shown community appetite for portable on-device inference. What Google appears to be betting on is that an Android-native, vendor-neutral runtime — backed by the same team that ships TensorFlow and Gemini — can become the default substrate for shipping AI features at Play Store scale.

For developers, the practical implication is that experimenting with NPU acceleration may soon require less vendor lock-in and less bespoke code. Whether LiteRT can deliver consistent performance across heterogeneous silicon, and how transparently it handles fallback when a given operator is not supported on a particular NPU, will likely determine adoption. Those questions are exactly what an early access program is positioned to answer, and the results could meaningfully shape how on-device AI is built on Android over the next year.