Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
AIArticle text · 659 words · 21 segments analyzed
| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) | About vLLM TorchTPU is the PyTorch-native TPU backend for vLLM.
It is an out-of-tree vLLM platform plugin, packaged as vllm-torchtpu, that runs vLLM's standard PyTorch model definitions on Google Cloud TPUs through TorchTPU (torch_tpu, via torch.compile(backend="tpu")) and a library of custom JAX/Pallas kernels for attention, Mixture-of-Experts, and quantized matmul. The goal is simple: the same vLLM you already use — same CLI, same OpenAI-compatible server, same model definitions — running at TPU speed with no model code changes.
PyTorch first.
Models run through vLLM's native PyTorch runner; no re-implementation in another framework is required to serve on TPU.
TPU-tuned kernels.
Ragged Paged Attention, dense and sparse MLA, Gated DeltaNet, fused MoE (including SparseCore offload), and FP8/INT8/MXFP4 quantized matmuls written in Pallas.
Scale-out built in.
TP, DP, PP, EP, and prefill/decode context parallelism compose across single-host and multi-host TPU slices, with disaggregated prefill/decode serving.
Standard vLLM experience.
Speculative decoding, prefix caching, multi-LoRA, structured output, and multimodal inputs work through the normal vLLM flags.
NoteLooking for the JAX backend?
The original TPU backend for vLLM, tpu-inference, has been rebranded as vLLM JAX.
It remains available for serving pure JAX model implementations on TPU. vllm-torchtpu is the recommended path for serving PyTorch model definitions on TPU.
Supported hardware TPU generation Status TPU v7x (Ironwood) Recommended — validated in CI on every pull request and nightly TPU v6e / v5e / v5p Experimental — not covered by CI See Cloud TPU Setup for project setup, quota, and provisioning commands. Get started Install vllm-torchtpu requires Python 3.12 on a Cloud TPU VM.
Install vllm-torchtpu from PyPI and a matching vLLM with a single command: pip install \ --extra-index-url https://download.pytorch.org/whl/cpu \ vllm-torchtpu Or pull the pre-built container from Docker Hub (vllm/vllm-torchtpu): docker pull vllm/vllm-torchtpu:latest For uv, editable source installs, cross-repo development against a local vllm checkout, and Docker images, see the full Installation guide and Docker guide. Serve a model Start an OpenAI-compatible server exactly as you would on any other vLLM platform: vllm serve meta-llama/Llama-3.2-1B-Instruct \ --tensor-parallel-size 1 \ --max-model-len 4096 Then send a request: curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.2-1B-Instruct", "messages": [{"role": "user", "content": "Explain how TPUs accelerate matrix multiplication."}], "max_tokens": 128 }' TipThe first startup compiles TPU graphs for the configured shapes and can take several minutes. Set VLLM_XLA_CACHE_PATH to persist compiled artifacts across restarts — see Compilation & Warmup Cache. The Quickstart covers offline batch inference with vllm.LLM, multi-device serving with TP/EP, and multimodal examples. Supported models and features vllm-torchtpu can serve most PyTorch models supported by vLLM, with first-class TPU kernel, quantization, and parallelism support for Qwen3 / Qwen3.5, DeepSeek-V3 / R1 / V4, Kimi-K2.5 / K3, GLM-5, Llama 3 / 4, GPT-OSS, Gemma 3 / 4, Mixtral, Qwen-VL and Gemma VL families, and diffusion models via vllm-omni. Supported Models & Features — validated model families, feature status, and quantization formats. Nightly support matrices — TPU v7x unit, correctness, and benchmark status, updated automatically from nightly CI.
Recipes and deployment TPU v7x (Ironwood) serving recipes TPU v6e (Trillium) serving recipes Kubernetes deployment Multi-host serving · Parallelism & scaling · Disaggregated serving (P/D) Quantization · Speculative decoding 🤝 Contribute We welcome contributions of all kinds — bug reports, model and feature requests, documentation fixes, and code.
🐞 Report bugs or request features: open a new issue.
👀 Review pull requests: help us keep quality high by reviewing open PRs.
💻 Write code: start with the Contributing Guide, which covers environment setup, DCO sign-off, the ready-label CI gate, and the PR lifecycle. Good first issues are a great place to begin.
Developer references: Testing & CI · Profiling · Benchmarking · Coding guidance · Optimization best practices 💬 Contact us For technical questions and feature requests, open a GitHub issue.
For discussion with fellow users, use the TPU support topic on the vLLM Forum. For coordinating contributions and development, join the vLLM Developer Slack (#sig-tpu).