Giampaolo Rodola: From Python 3.3 to today: ending 15 years of subprocess polling
Waiting for processes to terminate on POSIX systems using the busy-loop polling approach has been inefficient. This method constantly wakes the CPU to check process status, wasting cycles and increasing latency. The article introduces event-driven waiting as a superior alternative. Linux systems now support pidfd_open since kernel 5.3, allowing select, poll, or epoll to monitor process PIDs efficiently. This eliminates busy-looping by having the kernel wake the process only when the target exits or a timeout occurs. BSD and macOS systems utilize the kqueue syscall for similar event-driven process monitoring. Windows already employs an efficient event-driven method with WaitForSingleObject. The new implementations gracefully fall back to busy-looping if the event-driven mechanisms fail. Measurements show a significant reduction in context switches with the event-driven approach. Both poll and kqueue place the process in an efficient sleeping state, consuming zero CPU. This improvement, initially implemented in psutil, has now been contributed to Python's standard library subprocess module. This marks a significant return of an improvement from psutil to the standard library over 15 years later.
pidfd_opensince kernel 5.3, allowingselect,poll, orepollto monitor process PIDs efficiently. This eliminates busy-looping by having the kernel wake the process only when the target exits or a timeout occurs. BSD and macOS systems utilize thekqueuesyscall for similar event-driven process monitoring. Windows already employs an efficient event-driven method withWaitForSingleObject. The new implementations gracefully fall back to busy-looping if the event-driven mechanisms fail. Measurements show a significant reduction in context switches with the event-driven approach. Bothpollandkqueueplace the process in an efficient sleeping state, consuming zero CPU. This improvement, initially implemented inpsutil, has now been contributed to Python's standard librarysubprocessmodule. This marks a significant return of an improvement frompsutilto the standard library over 15 years later.