The /tmp/in
, /tmp/out
and /tmp/err
are named pipes, already created and opened by some process (for reading, writing and writing, respectively).
I would like to create a new process that pipes its stdin into /tmp/in
, and writes the contents of /tmp/out
to its stdout, and the contents of /tmp/err
to its stderr as they become available. Everything should work in a line buffered fashion. The process should exit when the other process, that created /tmp/in
, stops reading and closes /tmp/in
. The solution should work on Ubuntu, preferably without installing any extra package. I would like to solve it in a bash script.
mikeserv pointed out that without an SSCCE, it is hard to understand what I want. So, below is an SSCCE, but keep in mind that it is a minimal example so it is pretty silly.
The original setup
A parent process launches a child process and communicates with it through the child's stdin and stdout line-by-line. If I run it, I get:
$ python parent.py
Parent writes to child: a
Response from the child: A
Parent writes to child: b
Response from the child: B
Parent writes to child: c
Response from the child: C
Parent writes to child: d
Response from the child: D
Parent writes to child: e
Response from the child: E
Waiting for the child to terminate...
Done!
$
parent.py
from __future__ import print_function
from subprocess import Popen, PIPE
import os
child = Popen('./child.py', stdin=PIPE, stdout=PIPE)
child_stdin = os.fdopen(os.dup(child.stdin.fileno()), 'w')
child_stdout = os.fdopen(os.dup(child.stdout.fileno()))
for letter in 'abcde':
print('Parent writes to child: ', letter)
child_stdin.write(letter+'\n')
child_stdin.flush()
response = child_stdout.readline()
print('Response from the child:', response)
assert response.rstrip() == letter.upper(), 'Wrong response'
child_stdin.write('quit\n')
child_stdin.flush()
print('Waiting for the child to terminate...')
child.wait()
print('Done!')
child.py, must be executable!
#!/usr/bin/env python
from __future__ import print_function
from sys import stdin, stdout
while True:
line = stdin.readline()
if line == 'quit\n':
quit()
stdout.write(line.upper())
stdout.flush()
The desired setup and a hackish solution
Neither the parent's source file nor the child's source file can be edited; it is not allowed.
I rename the child.py to child_original.py (and make it executable). Then, I put a bash script (a proxy or a middle man if you wish) called child.py, start the child_original.py myself before running python parent.py
and have the parent.py call the fake child.py which is now my bash script, forwarding between the parent.py and the child_original.py.
The fake child.py
#!/bin/bash
parent=$$
cat std_out &
(head -n 1 shutdown; kill -9 $parent) &
cat >>std_in
The start_child.sh
to start child_original.py before executing the parent:
#!/bin/bash
rm -f std_in std_out shutdown
mkfifo std_in std_out shutdown
./child_original.py <std_in >std_out
echo >shutdown
sleep 1s
rm -f std_in std_out shutdown
The way of executing them:
$ ./start_child.sh &
[1] 7503
$ python parent.py
Parent writes to child: a
Response from the child: A
Parent writes to child: b
Response from the child: B
Parent writes to child: c
Response from the child: C
Parent writes to child: d
Response from the child: D
Parent writes to child: e
Response from the child: E
Waiting for the child to terminate...
Done!
$ echo
[1]+ Done ./start_child.sh
$
This hackish solution works. As far as I know, it does not meet the line buffered requirement and there is an extra shutdown fifo to inform the start_child.sh that child_original.py has closed the pipes and start_child.sh can safely exit.
The question asks for an improved fake child.py bash script, meeting the requirements (line buffered, exits when the child_original.py closes any of the pipes, does not require an extra shutdown pipe).
Stuff I wish I had known:
- If a high-level API is used for opening a fifo as a file, it must be opened for both reading and writing, otherwise the call to
open
already blocks. This is incredibly counter-intuitive. See also Why does a read-only open of a named pipe block? - In reality, my parent process is a Java application. If you work with an external process from Java, read the stdout and stderr of the external process from daemon threads (call
setDamon(true)
on those threads before starting them). Otherwise, the JVM will hang forever, even if everybody is done. Although unrelated to the question, other pitfalls include: Navigate yourself around pitfalls related to the Runtime.exec() method. - Apparently, unbuffered means buffered, but we don't wait until the buffer gets full but flush it as soon as we can.
Best Answer
If you get rid of the killing and shutdown stuff (which is unsafe and you may, in an extreme, but not unfathomable case when
child.py
dies before the(head -n 1 shutdown; kill -9 $parent) &
subshell does end upkill -9
ing some innocent process), thenchild.py
won't be terminating because yourparent.py
isn't behaving like a good UNIX citizen.The
cat std_out &
subprocess will have finished by the time you send thequit
message, because the writer tostd_out
ischild_original.py
, which finishes upon receivingquit
at which moment it closes itsstdout
, which is thestd_out
pipe and thatclose
will make thecat
subprocess finish.The
cat > std_in
isn't finishing because it's reading from a pipe originating in theparent.py
process and theparent.py
process didn't bother to close that pipe. If it did,cat > stdin_in
and consequently the wholechild.py
would finish by itself and you wouldn't need the shutdown pipe or thekilling
part (killing a process that isn't your child on UNIX is always a potential security hole if a race condition caused due to rapid PID recycling should occur).Processes at the right end of a pipeline generally only finish once they're done reading their stdin, but since you're not closing that (
child.stdin
), you're implicitly telling the child process "wait, I have more input for you" and then you go kill it because it does wait for more input from you as it should.In short, make
parent.py
behave reasonably:And your
child.py
can be as simple as(Note that I got rid of that fd dup calls because otherwise you'd need to close both
child.stdin
and thechild_stdin
duplicate).Since
parent.py
operates in line-oriented fashion, gnucat
is unbuffered (as mikeserv pointed out) andchild_original.py
operates in a line oriented fashion, you've effectively got the whole thing line-buffered.Note on Cat: Unbufferred might not be the luckiest term, as gnu
cat
does use a buffer. What it doesn't do is try to get the whole buffer full before writing things out (unlike stdio). Basically it makes read requests to the os for a specific size (its buffer size), and writes whatever it receives without waiting to get a whole line or the whole buffer. (read(2) can be lazy and give you only what it can give you at the moment rather than the whole buffer you've asked for.)(You can inspect the source code at http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/cat.c ;
safe_read
(used instead of plainread
) is in thegnulib
submodule and it's a very simple wrapper around read(2) that abstracts awayEINTR
(see the man page)).