Accelerating AI inferencing with external KV Cache on Managed Lustre

The demand for AI inference infrastructure is rapidly growing, with spending on inference expected to soon exceed training investments. This surge is fueled by the need for richer user experiences, larger context windows, and the rise of agentic AI. Efficient inference resource management is crucial for organizations aiming to enhance user experience and optimize costs. An experimental study found that using external key-value (KV) caches on high-performance storage like Google Cloud Managed Lustre can reduce total cost of ownership (TCO) by up to 35%. This is achieved by offloading prefill compute to I/O, allowing organizations to serve workloads with 43% fewer GPUs. The KV Cache is an optimization for Transformer-based LLMs that stores computed Key and Value vectors from preceding tokens, avoiding redundant calculations and speeding up inference. When KV caches exceed the capacity of host memory, external storage solutions become necessary, especially for large contexts and concurrent users. Agentic AI, designed for proactive problem-solving, further increases context lengths and KV cache sizes, exacerbating management challenges. Google Cloud Managed Lustre offers a high-throughput parallel file system ideal for large-scale, multi-node inference workloads exceeding host machine capacity. Experiments show significant performance improvements, including a 75% increase in inference throughput and a 44% reduction in mean time to first token when using Managed Lustre compared to host memory alone. This external KV cache solution provides a compelling TCO advantage over memory-only approaches by enabling more efficient utilization of expensive compute resources.