Mount Mayhem at Netflix: Scali... Note

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Netflix faced container scaling bottlenecks during their container runtime modernization. The issue arose because the new runtime used user namespaces for enhanced container security, employing kernel idmap features for efficient resource allocation, triggering frequent mount operations. These mount operations heavily contended for global kernel locks, particularly affecting nodes with many container layers. This contention was amplified on r5.metal instances, which have dual NUMA architecture. Benchmarks revealed instance type differences, with older Intel architectures showing higher container launch times. The bottleneck was rooted in Linux VFS path lookup code and was linked to the kernel's mount table's global locking. NUMA architecture amplified the problem due to memory access latency. Disabling hyperthreading improved performance by reducing resource competition. Centralized cache architectures in some CPUs exacerbated lock contention due to a single queueing structure. Distributed cache architectures, like those in AMD chips, showed better performance as contention was spread across multiple domains. Microbenchmarks validated the impact of NUMA, hyperthreading, and CPU architecture.
CdXz5zHNQW_w2NFMfjNQ5.png