Google Cloud Open-Sources TorchTPU: Native PyTorch on Google TPU Hardware with Day-Zero vLLM and SGLang Support

Background and the Problem

Google's Tensor Processing Units (TPUs) have consistently demonstrated superior compute density and interconnect bandwidth for large-scale training and inference. However, TPU adoption has historically been hampered by software friction: while JAX flourished, the dominant PyTorch ecosystem relied on PyTorch/XLA. This bridge suffered from crippling trade-offs: massive initial graph compilation overhead, poor support for dynamic shapes common in generative dialog, and severe friction when integrating custom Triton or CUDA kernels. Consequently, leading serving frameworks like vLLM and SGLang could not effortlessly unleash peak hardware throughput on TPUs. To eradicate this divide, Google Cloud officially open-sourced TorchTPU (@googlecloud).

Architecture and How It Works

1. Native Hardware Interface for PyTorch:

• Replaces cumbersome XLA graph compilation with direct tensor abstractions executing against TPU hardware drivers;
• Enables full PyTorch eager execution, allowing developers to inspect tensor values and step through execution interactively;

1. Triton & Custom Operator Compatibility:

• Enables native compilation of Triton kernels and custom C++ routines directly on TPU architectures, accelerating PagedAttention and MoE sparse dispatch;

1. Day-Zero Production Serving Integrations:

• vLLM and SGLang released official TorchTPU backends on Day 0, extending continuous batching and high-performance KV-cache management to TPU clusters;

1. Vertex Model Garden Expansion:

• Google Cloud simultaneously onboarded OpenAI models and Codex into its Model Garden, establishing Google Cloud as the only hyperscaler natively hosting all three frontier closed providers: Anthropic, OpenAI, and Google.

Benchmarks and Measured Results

Benchmarks on TPU v5p and TPU v6e instances confirmed dramatic efficiency improvements:

• Zero Graph-Compilation Stalls:
• Eliminates multi-minute initial compilation stalls and dynamic-shape re-compilations, delivering sub-millisecond dispatch responsiveness;
• 2.3x Throughput Acceleration:
• In vLLM benchmarking on 70B models, TorchTPU achieved a 2.3x throughput gain over legacy PyTorch/XLA while boosting KV-cache utilization by 38%;
• Infrastructure TCO Reduction:
• Delivers up to a 42% cost reduction per million tokens on TPU v6e compared to equivalent GPU deployments at matched P99 time-to-first-token (TTFT).

Getting Started for Developers

• Quickstart Installation:

  pip install torch-tpu vllm-tpu
  python -c "import torch, torch_tpu; print('TPU Ready:', torch.tpu.is_available())"
  

• Infrastructure Strategy: Enterprise AI platform teams should benchmark TorchTPU on Google Cloud TPU v6e instances to capture significant cost and latency dividends without sacrificing PyTorch developer velocity.