TorchTPU is now open source: R... Note

TorchTPU is now open source: Run native PyTorch on TPUs

PyTorch Conference 2026 introduced new products and integrations for running PyTorch on Google Cloud and other infrastructures. TorchTPU, the native PyTorch stack for Google TPUs, is now open source on GitHub. This allows seamless training, serving, and deployment of models on TPUs using familiar tools like vLLM, SGLang, and Hugging Face. The integration provides users with a choice of computing options including TPUs, GPUs, and CPUs with a consistent operational model. TorchTPU enables running existing PyTorch workloads on TPUs with minimal code changes and familiar APIs. It supports eager mode debugging and optimizes TPU utilization and memory through torch.compile. Hugging Face models now run natively on TorchTPU, simplifying deployment with device_map="tpu". Inference optimization is achieved through TorchTPU integrations with serving engines and fleet routing across TPUs and GPUs. vLLM provides high-throughput serving and memory efficiency on TPUs with features like continuous batching and quantization. SGLang accelerates agentic and structured workloads on TPUs by reusing KV cache across requests. llm-d 1.0, a CNCF Sandbox project, connects PyTorch serving engines with Kubernetes for distributed LLM inference. TorchTitan offers PyTorch-native distributed high-performance training at scale, validated on thousands of TPUs. VeRL, a popular RL framework, now includes a TPU backend for scalable post-training and reinforcement learning workloads. Ray has added native TorchTPU support, enabling orchestration of AI pipelines across various accelerators within a single workflow. LiteRT, Google's on-device AI inference solution, now joins the PyTorch ecosystem, allowing PyTorch models to run on diverse devices. The TPU agent suite is also in early preview on GitHub to automate onboarding, diagnostics, and performance tuning for PyTorch models on TPUs.
CdXz5zHNQW_YCinlopqHk.jpeg