Google Releases LiteRT Tensor API to Simplify Cross-Platform On-Device AI Optimization
Google's new C++ LiteRT Tensor API simplifies on-device ML model building and optimization across CPUs, GPUs, and NPUs, offering cross-platform support and zero-copy buffers.
Google has officially released the LiteRT Tensor API, a lightweight C++ interface designed to help developers construct, customize, and optimize machine learning models directly for on-device execution. The launch aims to streamline local AI deployment by marrying the flexibility of native C++ model creation with broad cross-platform portability across diverse hardware architectures.
Overcoming On-Device Hardware Fragmentation
Traditionally, developers seeking maximum control over local model performance had to manually write target computations directly in C++. While this provided low-level control, it resulted in code tightly bound to specific graphics and compute backends like Metal, OpenCL, or Vulkan. Consequently, target applications required extensive rewrites whenever deployed to new hardware platforms. In addition, handling zero-copy buffer interoperability and managing execution synchronization across CPUs, GPUs, and NPUs—while tying together pre-processing, inference, and post-processing steps—demanded heavy manual engineering.
The LiteRT Tensor API tackles these portability and execution bottlenecks by allowing engineers to construct computation graphs operation by operation using standard C++. Models defined through this approach run via LiteRT across CPU, GPU, and NPU hardware targets without requiring developers to rewrite code for each specific platform.
To simplify deployment pipelines, the API provides several core features:
- Direct weight ingestion: Developers can load model weights directly from standard checkpoints, including Hugging Face .safetensors files.
- Zero-copy buffer interoperability: Eliminates unnecessary data copying across accelerator memory during pre-processing, inference, and post-processing stages.
- AI coding agent integration: The C++ graph representation allows automated AI coding tools to inspect, alter, validate, and benchmark model architectures directly.
Leveraging AI Coding Agents for Optimization
To demonstrate the API's capabilities, Google utilized an AI coding agent to build a real-world application focused on multi-object segmentation and tracking using the SAM 2 model. The same C++ codebase was compiled and benchmarked across multiple devices: a macOS laptop using Metal, an iOS smartphone using Metal, and an Android phone using OpenCL.
During performance testing on a MacBook Pro, the SAM 2 implementation (utilizing fp16 precision across its image encoder and mask decoder) executed roughly 1.5 times faster than Apple's MLX framework.
Much of this efficiency stemmed from optimizations performed automatically by the AI coding agent. Because the model graph was written in plain C++, the agent moved SAM 2's per-frame logical decisions—such as determining whether a tracked object remains visible—out of traditional CPU code and directly into the execution graph. Furthermore, the agent identified operations that typically caused CPU fallbacks on WebGPU and rewrote them using GPU-compatible arithmetic.
Laurence Moroney, Director of AI at Arm, commented on the release via LinkedIn, pointing out the practical impact of defining model graphs in standard C++. Moroney noted that using an AI agent to iterate on structured primitives boosted performance from under 10 frames per second to over 30 frames per second at 512x512 resolution on WebGPU.
What it means for developers
The introduction of the LiteRT Tensor API offers mobile and edge developers a unified, low-level pathway to optimize models without getting locked into specific backend graphics APIs. However, developers do not necessarily need to rewrite every pipeline in C++ immediately. As Laurence Moroney recommended, engineers should generally start with standard model conversion techniques, reserving the LiteRT Tensor API for scenarios where fine-grained, graph-level control is essential or where keeping pre-processing and post-processing tasks on the hardware accelerator is required.
For engineers building hybrid architectures that combine edge processing with cloud endpoints, developers can try top AI models cheaply through one API at https://apixoai.online to complement local execution workloads.
Looking ahead, Google has confirmed several ongoing enhancements to the LiteRT Tensor API. Planned updates include direct support for the ML Drift backend, additional operations tailored for NPUs, automatic full-graph quantization, and broader dynamic-shape capabilities to accommodate varying image dimensions, batch sizes, and sequence lengths.
Source: Google Launches LiteRT Tensor API for Building, Customizing, and Optimizing ML Models for On-Device — InfoQ AI & ML. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
