Giampaolo Rodola: From Python 3.3 to today: ending 15 years of subprocess polling

Waiting for processes to terminate on POSIX systems using the busy-loop polling approach has been inefficient. This method constantly wakes the CPU to check process status, wasting cycles and increasing latency. The article introduces event-driven waiting as a superior alternative. Linux systems now support `pidfd_open` since kernel 5.3, allowing `select`, `poll`, or `epoll` to monitor process PIDs efficiently. This eliminates busy-looping by having the kernel wake the process only when the target exits or a timeout occurs. BSD and macOS systems utilize the `kqueue` syscall for similar event-driven process monitoring. Windows already employs an efficient event-driven method with `WaitForSingleObject`. The new implementations gracefully fall back to busy-looping if the event-driven mechanisms fail. Measurements show a significant reduction in context switches with the event-driven approach. Both `poll` and `kqueue` place the process in an efficient sleeping state, consuming zero CPU. This improvement, initially implemented in `psutil`, has now been contributed to Python's standard library `subprocess` module. This marks a significant return of an improvement from `psutil` to the standard library over 15 years later.