Skip to content

Home

vLLM TPU vLLM TPU

| Documentation | Blog | User Forum | Developer Slack |

About

vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project. The new backend now provides a framework for developers to:

  • Push the limits of TPU hardware performance in open source.
  • Provide more flexibility to JAX and PyTorch users by running PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX.
  • Retain vLLM standardization: keep the same user experience, telemetry, and interface.

Although vLLM TPU’s new unified backend makes out-of-the-box high performance serving possible with any model supported in vLLM, the reality is that we're still in the process of implementing a few core components.

For this reason, we’ve provided a Recommended Models and Features page detailing the models and features that are validated through unit, integration, and performance testing.

Getting Started

If you are new to vLLM on TPU, we recommend starting with the Quickstart guide. It will walk you through the process of setting up your environment and running your first model. For more detailed installation instructions, you can refer to the Installation guide.

Compatible TPU Generations

  • Recommended: v5e, v6e
  • Experimental: v3, v4, v5p

Check out a few v6e recipes in the tpu-recipes repository!

Developer Guides

If you are interested in contributing to the project or want to learn more about the internals, check out our developer guides:

Contribute

We're always looking for ways to partner with the community to accelerate vLLM TPU development. If you're interested in contributing to this effort, check out the Contributing guide and Issues to start. We recommend filtering Issues on the good first issue tag if it's your first time contributing.

Contact us