Building Pinterest’s VLM Servi... Note

Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo

Pinterest leverages Vision-Language Models (VLMs) to enhance its visual search and discovery platform. These models are crucial for powering new features like Pinterest Assistant and hybrid search. Pinterest customizes open-source VLMs to meet its specific product and scale requirements. Serving VLMs presents unique challenges beyond traditional text-only LLMs, including handling multiple images and larger KV caches. To address these, Pinterest built its VLM serving stack on NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs offer superior compute throughput and memory capacity for demanding AI workloads. Dynamo provides a distributed orchestration layer for optimizing multimodal inference across the entire serving path. Request payloads for VLMs are more complex, requiring image downloading, preprocessing, and prompt construction. The prefill stage in VLM serving is often dominant and computationally expensive, necessitating advanced cache management. Multi-turn VLM interactions further complicate serving due to evolving visual context and heavier prefill demands. Managing KV cache memory pressure is critical, requiring strategies like KV-aware routing and offloading. Pinterest's solution utilizes disaggregated prefilling and decoding, KV cache offloading with LMCache, and multimodal support within Dynamo and vLLM. They also implemented custom projection embeddings to reduce the cost of processing large visual contexts. This optimized VLM serving stack significantly improves performance, demonstrating substantial gains in Time-to-First-Token.
CdXz5zHNQW_ydHVPNTidy.png