Google and Meta Release TorchTPU to Bring Native PyTorch Support to TPUs
Google and Meta have introduced TorchTPU, a native backend designed to let PyTorch models run directly on Google TPUs without complex code alterations.
At the PyTorch Conference North America 2026 in San Jose, Google and Meta jointly unveiled TorchTPU, an open-source native backend designed to run PyTorch workloads directly on Google's Tensor Processing Units (TPUs). The project marks a major shift away from PyTorch/XLA, a bridge introduced in 2020 that often necessitated intrusive source-code modifications.
With TorchTPU, the two tech giants aim to make switching compute backends as simple as updating code from model.to("cuda") to model.to("tpu"). By integrating directly into PyTorch's low-level dispatcher via the PrivateUse1 extension point, TorchTPU elevates Google's custom silicon alongside GPUs and CPUs as a fully native target.
Meta software engineer Zongwei Zhou, who co-presented the announcement with Google Cloud VP of engineering Bill Jia, explained that Meta intends to transition larger-scale large language model workloads onto TPUs using the new integration. Jana van Greunen, director of PyTorch engineering at Meta, noted that operating at the frontier of AI requires immense compute scale and diverse hardware, stating that the company cannot afford to remain tied to a single architecture.
Shifting Workloads and Surging Token Demand
The push for hardware diversity coincides with rapid growth in operational demands. PyTorch Foundation Executive Director Mark Collier reported that inference has officially surpassed training as the largest workload across data centers, sparking substantial demand for compute capacity. Collier highlighted data demonstrating the rapid pace of efficiency gains in the sector: DeepSeek's Flash model became 60% cheaper in 37 days, while Kimi K3 saw a 68% speedup on identical hardware within 33 days.
Agent-driven workloads are driving much of that consumption. According to Mazin Gilbert, executive director of the Agentic AI Foundation, public figures from OpenRouter indicate that agent token usage overtook human consumption on February 6, 2026. Since that crossover, agent demand has surged 14-fold, far outpacing the 2.8-fold growth seen from human users.
To accommodate these scaling demands, Google and Meta open-sourced four primary repositories: TorchTPU, vLLM-TorchTPU, SGLang-TorchTPU, and TorchTitan. Popular open models from Hugging Face run without custom modifications, and orchestration tools like Ray and llm-d provide multi-host scaling on Kubernetes. During benchmark presentations, Jia highlighted near-linear scaling across more than 8,000 TPU chips, including a demonstration running a Qwen3 235B mixture-of-experts model on 8,192 TPU v7x chips that achieved 28% model FLOPs utilization.
Hardware Expansion Across the Industry
Google is not alone in seeking deeper PyTorch integration for proprietary chips. Amazon Web Services placed TorchNeuron, a native backend for its Trainium hardware, into public beta on October 20. Like TorchTPU, it relies on PyTorch's PrivateUse1 extension to support .to('neuron'). However, initial testing shows early-stage limitations: The Next Web reported that TorchNeuron skips roughly 22,000 PyTorch tests and lacks dynamic input shape support.
Industry analysts note that while software portability has improved, broader adoption still faces practical hurdles. While PyTorch provides a standardized interface across architectures, low-level optimization layers and hardware supply chains remain under vendor control.
What it means for developers
For engineering teams, TorchTPU significantly reduces the friction required to test models on Google Cloud TPUs. Because modern serving engines such as vLLM and SGLang now run natively on the backend, teams can experiment with alternatives to traditional GPU clusters without rebuilding their training or inference software stacks.
However, Google cautioned that replacing the device string is primarily an entry point. Because operations, sharding configurations, and tensor shapes are often heavily tuned for GPU architectures, reaching maximum hardware efficiency on TPUs still demands dedicated profiling and optimization. To help automate onboarding and performance tuning, Google released a preview of its TPU agent suite on GitHub.
Teams that want to evaluate output performance before committing engineering resources to specialized hardware clusters can also look at consolidated API layers. Developers can try top AI models cheaply through one API at https://apixoai.online to assess different model architectures before making major hardware migration plans.
Ultimately, TorchTPU brings PyTorch closer to a truly hardware-agnostic runtime. While teams must still budget engineering time for sharding and kernel adjustments, the ability to target TPUs natively offers a viable path around hardware bottlenecks.
Source: PyTorch Conference 2026: Google and Meta Open-Source TorchTPU for Native PyTorch on TPUs — InfoQ AI & ML. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
