Netflix TechBlog | Medium
Follow
Investigation of a Cross-regional Network Performance Issue
Netflix operates a vast cloud infrastructure on Amazon AWS across multiple regions to facilitate seamless content delivery. When an application experienced cross-region data synchronization issues, initial suspicion fell on the network due to timeouts.Upon investigation, the network engineers discovered TCP RST packets originating from the client, indicating that the client was terminating connections prematurely.Packet captures confirmed that the server was continuously sending data while the client abruptly ended connections after 30 seconds.Examining the application revealed that it had a 30-second timeout for receiving initial data from servers, leading it to terminate connections if data wasn't received within that time frame.Despite no recent changes to the application or server, the issue coincided with a Linux kernel upgrade from version 6.5.13 to 6.6.10.Methodical kernel bisection revealed a commit responsible for the issue, which had introduced a new mechanism for calculating TCP receive window size.The original calculation assumed 50% overhead, while the new mechanism dynamically calculated overhead based on actual data characteristics, resulting in a smaller receive window.With a smaller window, the application could not receive data quickly enough within the 30-second timeout, leading to connection terminations.The root cause was the obsolete sysctl_tcp_adv_win_scale parameter, which was replaced with a more accurate scaling_ratio to calculate the receive window size.Rolling back the kernel upgrade restored normal operation to the application, demonstrating that the kernel change was the underlying cause of the issue.