Google Open-Sources ML Drift to Accelerate On-Device GPU Inference

Desktop execution across Windows and Linux is facilitated through Dawn, Chromium’s WebGPU implementation, while macOS retains native Metal backend support. Dawn acts as a hardware abstraction layer, allowing developers to target WebGPU rather than writing separate platform implementations for DirectX or Vulkan.

In internal evaluations running Gemma models on developer-grade workstation systems, Google recorded a 12% decrease in memory overhead compared to alternative frameworks such as llama.cpp and MLX LM. Furthermore, the engine is already deployed at scale, handling workloads on hundreds of millions of client devices via Chrome, Google Meet, Google Photos, and YouTube Shorts.

Production metrics and ecosystem feedback

Production deployments demonstrate measurable performance gains across consumer applications. Within YouTube Shorts, ML Drift cuts average frame latency by up to 40% for segmentation effects on iOS and Android. In Google Photos, computational photography pipelines and segmentation tasks execute up to two seconds faster.

Third-party applications have reported similar improvements. Adobe observed up to a 30% performance boost in features such as Select Subject, Select Sky, and Adaptive Portrait within Lightroom and Photoshop. On Android, Snapchat achieved a 30% decrease in model latency for machine-learning-driven face and style effects, making it practical to run larger diffusion models on mobile hardware.

Industry engineers have noted the architectural benefits of these additions. Commenting on the announcement, AI/ML engineer Pavithra Agraharam stated that native 5D tensor support without layout workarounds represents a major improvement for running spatial and transformer models on edge GPUs, noting that decoupling logical tensors from physical memory allocations helps mitigate memory fragmentation during active inference.

At the same time, some developers are looking for additional data. Matthew Charles Vladislav Busel, founder and developer of Tensorust, raised questions regarding how ML Drift compares with Apple's Core ML on recent iPhones in terms of latency and memory usage, pointing out that many production applications currently target Core ML on iOS devices. Google has not yet published comparative metrics against Core ML.

What it means for developers

With the launch of ML Drift, Google has confirmed it will cease updates to the legacy TFLite GPU delegate. Teams maintaining existing deployments are advised to transition to the LiteRT ML Drift accelerator to preserve backward compatibility and receive ongoing improvements.

For engineers building consumer applications, the LiteRT ML Drift accelerator provides a drop-in integration path. GPU and systems engineers requiring deeper control can utilize the lower-level C++ API to assemble custom GpuModel compute graphs and author specialized GPU shaders.

While running models on client hardware is ideal for low-latency tasks like camera filters or localized features, larger workloads often still depend on cloud infrastructure. For server-side tasks, developers can try top AI models cheaply through one API at https://apixoai.online, while deploying lighter models directly to user devices using engines like ML Drift.

Source: Google ML Drift Accelerates On-Device AI Inference Across Diverse GPUs — InfoQ AI & ML. Written by the Apixo team from that report.