Enhancing Netflix Reliability ... Note

Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding

Netflix has implemented a service-level load shedding technique to enhance user experience and system resilience. This technique prioritizes critical user-initiated requests over non-critical pre-fetch requests within the same service. By implementing a concurrency limiter with partitioning, PlayAPI, a critical backend service, allocates 100% throughput to critical requests and utilizes excess capacity for non-critical requests. This prioritization ensures that user-initiated requests maintain high availability even during traffic spikes or high latency. The success of this approach led to the development of a generic library for service prioritization, allowing services to define multiple priority levels. This library utilizes CPU utilization as a measure of system load, progressively shedding lower priority traffic to maintain user experience and provide more time for auto-scaling. Experiments have shown that this approach effectively sheds non-critical and critical traffic under extreme load conditions, maintaining reasonable latency and stable throughput. Netflix emphasizes the importance of avoiding anti-patterns such as no shedding or congestive fail, which can degrade user experience or lead to system instability.