Netflix TechBlog | Medium
Follow
Java 21 Virtual Threads - Dude, Where’s My Lock?
Netflix, known for its extensive use of Java, was enthusiastic about adopting virtual threads for their performance benefits. However, during the migration to Java 21, engineers encountered intermittent timeouts and unresponsive instances in applications using virtual threads with SpringBoot 3 and embedded Tomcat. Investigations revealed a surge in sockets stuck in the closeWait state, indicating the applications were failing to close connections. Thread dumps initially showed idle JVMs, but further analysis using specific commands revealed a large number of "blank" virtual threads, which are threads created but not yet running. This pointed to a problem with virtual thread execution, where Tomcat was creating a new virtual thread for each incoming request, but the underlying operating system threads were all occupied. Deeper analysis of the thread dumps showed that these operating system threads were pinned to virtual threads stuck waiting to acquire a lock within a synchronized block in the code. The culprit was a lock held by a regular platform thread that was unable to reacquire the lock after completing a wait operation. This deadlock situation prevented new virtual threads from being scheduled, leading to the observed performance issues.