In stream processing and queuing, we have this notion of backpressure, which is that if a producer process is going faster than a consumer process, we should have a mechanism for slowing down the producer to avoid exceeding available memory/storage (without dropping messages, which may or may not be acceptable).
I've been curious for a little while now about whether stdio can be used to exert backpressure in this sense on a producer Unix process (e.g. foo in foo | bar). It would seem that even if writes to stdout were blocked when a buffer reached capacity, it would still be necessary for the producer process to Do The Right Thing (TM), and not accumulate data in memory waiting to be written to stdout. A single threaded, blocking program would seem to pass the test, but an asynchronous program may have to have it's own internal buffering and backpressure mechanism for in order not to explode with data waiting to be written.
So; to what extent is this possible and what are the particulars?
Best Answer
A pipe has a limited buffer size. If the producer goes ahead of the consumer, the data progressively fills the pipe's buffer. If the buffer is filled up, the
writecall in the producer blocks until there is room. So backpressure is built into the system.The size of the buffer is at least 512 bytes on any POSIX-compliant system, and often larger and potentially configurable on modern unices. See How big is the pipe buffer? for more details.