Shell – Why process does not (seem to) exist before fifo read

fifoio-redirectionprocessshell

I have a trivial Python script:

#!/usr/bin/python

import os, sys

sys.stderr.write('I am %s' % os.getpid())
sys.stderr.flush()

print "hello"

sys.stderr.write('I am done')

When I run this script from Bash and redirect the stdout to FIFO:

$ mkfifo fifo
$ /pyscript > fifo

the strange thing is, that before I read from the FIFO:

I won't get the "I am (PID)" message,
I can't see the script using ps -ef
and using lsof, I can't see anybody has the fifo file open!

Once I do read from the FIFO, the both messages that I'm writing to stderr appear at once.

What is happening?

The background: I'm writing a test where I create FIFO and write 'hello' to it; then I run a test and expect the SUT to not read from it; ie. just ignore the file. I tried to do mkfifo test_fifo; /bin/echo hello > test_fifo &; run_the_test; killall echo but to my surprise, the echo process never exists! How should I "clean up" after such test (apart from rm test_fifo)…?

Best Answer

The shell will not run a program until any file redirections have been completed successfully. The command pyscript > fifo will cause the shell to fork and then try to open fifo. This will succeed only when some process has opened fifo for reading. Until then, the child shell will be in pipe wait.

$ (sleep 10;ps -l -t pts/0)&
[1] 2574
$ pyscript > fifo
F S   UID    PID   PPID WCHAN  TTY    CMD
0 S  1000   2554   2535 wait   pts/0  bash  //interactive shell
1 S  1000   2574   2554 wait   pts/0  bash  //fork of shell, parent of sleep and ps
1 S  1000   2576   2554 pipe_w pts/0  bash  //fork of shell, waiting to open fifo
0 R  1000   2577   2574 -      pts/0  ps

Related Solutions

Bash – What are the difference between those four commands (fifo, process substitution, redirection…)

The key here is that opening a FIFO is a blocking operation. The open only returns once both ends are connected, i.e. once the fifo is open for both reading and writing.

man fifo(7)

Normally, opening the FIFO blocks until the other end is opened also.

In the 1st case, the shell forks to execute the pipeline, so opening the fifo for reading (cat fifo) and opening the fifo for writing (> fifo) happen in separate processes, so happen independently.

In the 2nd case, the open for reading and open for writing (3<>fifo) happen in a single step.

In the 3rd case, <(cat fifo) expands to a filename, e.g. /dev/fd/42. So it's like you're running nc -l localhost 8888 /dev/fd/42 > fifo. You need an extra < for it to be equivalent, e.g. nc -l localhost 8888 < <(cat fifo) > fifo.

In the 4th case, the shell is trying to open the fifo for reading (< fifo) and open it for writing (> fifo) as part of the same process. The shell does them one at a time, left to right. So it tries to open fifo for reading, and blocks forever, waiting for something to open fifo for writing. I think you'll find that in this case, nc never even got started, and the port was never opened for listening.

Bash – Communicating from bash with grunt shell

I now more on less settled on this version:

#!/bin/bash

in=/tmp/grunt_in
out=/tmp/grunt_out
err=/tmp/grunt_err

if [ ! -p $in ]
then
    mkfifo $in
    mkfifo $out
    mkfifo $err
    { script -q -c "pig 1>$out 2>$err" <>$in; rm $in $out $err; } &
fi

{
    echo "fs $*"
    echo
    echo "-- end"
} >$in
cat $err >&2 &
catpid=$!
sed -n -u '/^grunt> -- end/q;/^grunt>/d;p' <$out
kill $catpid

So I simply redirect stderr inside of the script command. I also replaced the round braces with curly braces and removed the disown because I didn't see any advantage in doing that. I also replaced $err with a FIFO in order to be able to output it early, but that also adds some complications to kill the cat.

This works fairly well so far except that when I truncate the output by piping through head I get a truncated or extra output in the next command. Apparently I need a way to properly flush the named pipes. I would be glad if someone has any hints.

Best Answer

Related Solutions

Bash – What are the difference between those four commands (fifo, process substitution, redirection…)

Bash – Communicating from bash with grunt shell

Related Question