Why pipes sometimes get "stuck... Note
Julia Evans

Why pipes sometimes get "stuck": buffering

Buffering is common in terminal programs to improve performance by grouping output until a certain size threshold is met. This can cause issues when data is added slowly to a pipe, as programs may buffer their output and never write it.Grep and similar programs default to block buffering when writing to pipes but use line buffering when writing to terminals, explaining why the command "tail -f /some/log/file | grep thing1 | grep thing2" may not display output.Several commands buffer their output, including grep, sed, awk, tcpdump, and jq, while commands like tail, cat, and tee do not.Programming languages like C, Python, Ruby, and Perl may also buffer output, with various methods to disable buffering.When Ctrl-C is pressed on a pipe, the program's output buffer may be lost, as the signal is received before the buffer can be flushed.Buffering also occurs when redirecting to a file, but it generally behaves as expected, with the contents of the buffer being written before the program exits.To avoid buffering, one can run a program that finishes quickly, use the "--line-buffered" flag with grep, rewrite the command using awk, use stdbuf, or use unbuffer to force the program to behave as if writing to a terminal.The ideal solution depends on the specific situation, with unbuffer being a reliable choice for its consistent behavior.While buffering is generally not a common issue, it can arise when data is slowly added to a pipe. An environment variable to disable buffering could be useful, but its design and implementation present challenges.