Problem with pipes. Pipe terminates when reader done

fifopipe

I am on OSX, using bash, trying to make sense of pipes. I wish to let a program communicate in two directions with a bash shell. I want to set this up in such a way that this is always the same shell, so that I can cd to some directory and bash will remember (rather than using a new bash shell all the time).

What I have tried so far is this. From a new terminal (A), do

mkdir /tmp/IOdir
cd /tmp/IOdir
mkfifo pToB
mkfifo bToP
tail -f -1 pToB | bash >> bToP

Then, to test this connection, I can do, from a new terminal (B)

cd /tmp/IOdir
echo echo hello > pToB

and from a third terminal (C)

cd /tmp/IOdir
(read myline &&  echo $myline) < bToP

This behaves how I want. The same bash shell stays active, and the output comes through on the other side. Call this state of affairs X, so that I can refer later to it.

Onward from state X

However, now, from this state X, we cannot do the same thing again. That is, if we do from terminal (B)

echo echo hello > pToB

and then from terminal C

(read myline &&  echo $myline) < bToP

Then nothing comes through in terminal C. Moreover, if we again do, from terminal B

echo echo hello > pToB

The bash shell closes.

What I could have done in state X was first do, from terminal C

(read myline &&  echo $myline) < bToP

and then from terminal B

echo echo hello > pToB

In this case hello comes through at terminal C, and it seems like we are in state X again. So we can basically repeat this forever. Now this might seem sufficient for two way communication, but my program is such that if it requests a new line like this

(read myline &&  echo $myline)

and there is no new line, it will "hang" (just like bash, in fact I mean to use a call to bash in the program). It is therefore not able to send input to pToB after that and there is nothing I can do.

Questions

Is there a way to set this up without doing too much programming in C? Is there a way to do this more elegantly without using two named pipes? What is causing the pipe to close in one scenario, and not in the other?

Edits

From this page on wikipedia, we have

Full-duplex (two-way) communication normally requires two anonymous pipes.

On one hand, it seems at least I have the right number of pipes. On the other, I am using named pipes, not anonymous ones. So maybe this will be hard/impossible.

Furthermore mkfifo gnu/linux source is likely defined in terms of mknod gnu/linux source, which is also a unix command. But I'm not sure if much can be learned from that.

Here is an introduction to pipes in C, including the linux source of pipe. Maybe that can tell us why a pipe gets closed, if that is indeed what happens.

Here is a related question about preventing that fifos get closed. I tried tying the pipes to background sleeping processes as was done in an answer tere but that didn't help.

Best Answer

As for the cause, use strace.

tail -f | strace bash >> foo

The second echo echo hello > pToB gives me then this:

rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
read(0, "e", 1)                         = 1
read(0, "c", 1)                         = 1
read(0, "h", 1)                         = 1
read(0, "o", 1)                         = 1
read(0, " ", 1)                         = 1
read(0, "h", 1)                         = 1
read(0, "e", 1)                         = 1
read(0, "l", 1)                         = 1
read(0, "l", 1)                         = 1
read(0, "o", 1)                         = 1
read(0, "\n", 1)                        = 1
write(1, "hello\n", 6)                  = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=3299, si_uid=1000} ---
+++ killed by SIGPIPE +++

So, the second time it tries to write hello\n, it gets a broken pipe error; that's why you can't read hello (it was never written), and bash quits so that's the end of it.

You'd have to use something that keeps the pipe open, I guess.

How about this?

(while read myline; do echo $myline; done) < pToP

For more background information, man 7 pipe may be relevant, it describes the various error cases around pipes.

Related Solutions

Shell Pipes – How to Understand Pipes

About your performance question, pipes are more efficient than files because no disk IO is needed. So cmd1 | cmd2 is more efficient than cmd1 > tmpfile; cmd2 < tmpfile (this might not be true if tmpfile is backed on a RAM disk or other memory device as named pipe; but if it is a named pipe, cmd1 should be run in the background as its output can block if the pipe becomes full). If you need the result of cmd1 and still need to send its output to cmd2, you should cmd1 | tee tmpfile | cmd2 which will allow cmd1 and cmd2 to run in parallel avoiding disk read operations from cmd2.

Named pipes are useful if many processes read/write to the same pipe. They can also be useful when a program is not designed to use stdin/stdout for its IO needing to use files. I put files in italic because named pipes are not exactly files in a storage point of view as they reside in memory and have a fixed buffer size, even if they have a filesystem entry (for reference purpose). Other things in UNIX have filesystem entries without being files: just think of /dev/null or others entries in /dev or /proc.

As pipes (named and unnamed) have a fixed buffer size, read/write operations to them can block, causing the reading/writing process to go in IOWait state. Also, when do you receive an EOF when reading from a memory buffer ? Rules on this behavior are well defined and can be found in the man.

One thing you cannot do with pipes (named and unnamed) is seek back in the data. As they are implemented using a memory buffer, this is understandable.

About "everything in Linux/Unix is a file", I do not agree. Named pipes have filesystem entries, but are not exactly file. Unnamed pipes do not have filesystem entries (except maybe in /proc). However, most IO operations on UNIX are done using read/write function that need a file descriptor, including unnamed pipe (and socket). I do not think that we can say that "everything in Linux/Unix is a file", but we can surely say that "most IO in Linux/Unix is done using a file descriptor".

Bash – Races when piping two commands to a named pipe

Why not just do:

{ echo foo; echo bar;} > /tmp/p

If you want your controlling script to leave the pipe open, you can do:

exec 3<> /tmp/p

Opening a named pipe in read-write mode is to avoid that to block if the pipe hasn't been opened yet. That would instantiate it if it wasn't yet. It works on Linux at least but is not guaranteed to by POSIX.

Alternatively (and portably):

: < /tmp/p & exec 3> /tmp/p

Then instead of having each process open the named pipe, you can also do:

cmd >&3

And in the end, you'd do:

exec 3>&-

To close the writing end to let readers know it's finished.

change all <s to >s and <s to >s if you need the logic to be the other way round.

Best Answer

Related Solutions

Shell Pipes – How to Understand Pipes

Bash – Races when piping two commands to a named pipe

Related Question