Netflix TechBlog | Medium
Follow
Noisy Neighbor Detection with eBPF
Netflix uses eBPF to continuously monitor run queue latency, an indicator of noisy neighbors (containers heavily utilizing server resources, degrading performance in adjacent containers).eBPF hooks (sched_wakeup, sched_wakeup_new, sched_switch) are used to capture run queue latency and associate it with containers (cgroup IDs) using kfuncs (kernel functions) for safe RCU-protected data access.A rate limiter in eBPF balances observability with performance by limiting data points sent to userspace.Userspace processes events from the eBPF ring buffer and emits metrics to Atlas, including run queue latency (runq.latency) and preemption count (sched.switch.out) by cgroup ID.Both runq.latency and sched.switch.out metrics are necessary to identify noisy neighbors, as runq.latency alone can be misleading when containers are at their CPU limit.A case study demonstrates a noisy neighbor issue causing a spike in run queue latency and preemptions when a new container fully utilizes host CPUs.System processes were identified as the noisy neighbors using the sched.switch.out metric.Optimizations to the eBPF code, including the use of BPF_MAP_TYPE_HASH, direct task struct member access, and ignoring kernel tasks, minimize overhead.A Linux kernel patch was submitted and accepted to improve the calculation of statistics in the kernel.BPFtop, an open-source eBPF process monitoring tool, was used to measure the overhead of the eBPF code.