Giampaolo Rodola: From Python ... Note

Giampaolo Rodola: From Python 3.3 to today: ending 15 years of subprocess polling

Waiting for processes to terminate on POSIX systems using the busy-loop polling approach has been inefficient. This method constantly wakes the CPU to check process status, wasting cycles and increasing latency. The article introduces event-driven waiting as a superior alternative. Linux systems now support pidfd_open since kernel 5.3, allowing select, poll, or epoll to monitor process PIDs efficiently. This eliminates busy-looping by having the kernel wake the process only when the target exits or a timeout occurs. BSD and macOS systems utilize the kqueue syscall for similar event-driven process monitoring. Windows already employs an efficient event-driven method with WaitForSingleObject. The new implementations gracefully fall back to busy-looping if the event-driven mechanisms fail. Measurements show a significant reduction in context switches with the event-driven approach. Both poll and kqueue place the process in an efficient sleeping state, consuming zero CPU. This improvement, initially implemented in psutil, has now been contributed to Python's standard library subprocess module. This marks a significant return of an improvement from psutil to the standard library over 15 years later.