Netflix runs its entire large language model stack internally, managing deployment and inference directly. They integrated this into their existing production environment instead of a separate machine learning silo. This approach involved carefully selecting an inference engine, deciding how models would be packaged, designing the API surface, defining deployment strategies, and enforcing output constraints.The chosen inference engine is vLLM, selected for its operational fit, ability to load custom architectures, extensibility, debuggability, and familiarity among practitioners. Models are packaged using the vLLM backend in Triton to allow dynamic I/O tensor specification, promoting independent evolution of models and frontends. An OpenAI-compatible HTTP frontend was added alongside their existing gRPC interface for broader ecosystem compatibility.For deployment, Netflix uses a Versioned strategy to handle potential schema changes between model versions, allowing consumers to update independently. When model interfaces are stable, the less costly Red-Black deployment strategy is employed. One key operational challenge was slow model startup times, which they addressed by materializing large models on Amazon FSx for faster access.Observability was improved by creating a unified /metrics endpoint that aggregates data from both vLLM and Triton. A significant feature is constrained decoding at scale, implemented via vLLM's custom logits processor. This allows models to generate compliant outputs directly, avoiding post-inference correction.An initial pure-Python implementation of constrained decoding struggled with scaling due to the Global Interpreter Lock (GIL) and sequential processing on the CPU. This only became apparent under high concurrency, leading to significant tail latencies. The system relies on a Java control plane for managing deployment, versioning, and autoscaling. Both real-time and cached batch inference paths are supported by their serving system. This unified system handles the complete downstream consumer flow, including routing, candidate generation, and logging.
/metricsendpoint that aggregates data from both vLLM and Triton. A significant feature is constrained decoding at scale, implemented via vLLM's custom logits processor. This allows models to generate compliant outputs directly, avoiding post-inference correction.An initial pure-Python implementation of constrained decoding struggled with scaling due to the Global Interpreter Lock (GIL) and sequential processing on the CPU. This only became apparent under high concurrency, leading to significant tail latencies. The system relies on a Java control plane for managing deployment, versioning, and autoscaling. Both real-time and cached batch inference paths are supported by their serving system. This unified system handles the complete downstream consumer flow, including routing, candidate generation, and logging.