Communicate backwards in a pipe

pipestderrstdinstdiostdout

I have a simple pipeline:

node foo.js | node bar.js

bar.js will read from stdin to get data from foo.js.

But what I want to do is ensure that bar.js gets one of the last messages from foo.js before foo.js decides it's OK to exit. Essentially I want to create a simple request/response pattern.

foo writes to stdout –> bar reads from stdin –> how can bar send a
message back to foo?

Is there a way to communicate backwards in a pipeline or should there never be a need to do that?

Best Answer

No. A pipeline is a one-way communication channel. That's why it's called a "pipeline"; you couldn't send oil back up a pipeline if you tried, either.

However, if bar.js has to talk to foo.js too, you do have a few options:

Create a unix domain socket instead of a pipeline, and start both foo.js and bar.js separately (i.e., don't pipe the output of foo.js into bar.js anymore). I don't know how you do that from node, but essentially a unix domain socket is a network socket that uses filenames rather than IP addresses and works inside the kernel. Sockets are meant for bidirectional communication, but require more setup than a simple pipe (e.g., a listening socket can talk to multiple instances of bar.js). You may find unix domain sockets in the filesystem, but it's not strictly necessary (and indeed Linux allows creating a unix domain socket without a trace on the filesystem).
Use mkfifo to create a named pipe (or use some node API to create one, if that exists; again, I do not know node). Then, in foo.js, open that named pipe and read from it. Your bar.js script can open the same named pipe and write to it.

The latter will be easiest to transition to, since you still use file I/O (opening a named pipe requires opening a file on the filesystem), but would still be unidirectional (although you'd have two channels, one in each direction). The former is slightly cleaner and also allows you to more easily migrate one of the two scripts to a different host if that ever should become necessary.

At any rate, if your scripts are now communicating bidirectionally, then for clarity I would suggest you start them as separate processes, rather than having one process pipe into the other. IMHO, they're equal partners now, and your command line should show that. That's just a detail though, and certainly not technically required.

Related Solutions

Command that prints file contents given filename on stdin

After doing a bit more research I realised that this scenario is exactly what xargs is designed for:

./my-command args | cut -d : -f 5 | xargs cat

Which will transform the output of stdin into an invocation of cat with an actual filename and thus print out the file contents

Bash – Read from and write to file from stdin and stdout

The redirection operator to open a file in read+write mode without truncation is <> in all Bourne-like shells (that maps to open(file, O_RDWR|O_CREAT) (though zsh also throws in a O_NOCTTY) or fopen(file, "w+")):

exec 3<> "$file"

opens the $file on file descriptor 3 in read+write mode (without truncating it and creating it if it didn't exist).

However, only ksh93 and zsh have seeking operators. dd can seek, but not backwards. And note no shell except zsh can have NUL bytes in their variables.

In zsh:

zmodload zsh/system
exec 3<> $file
sysread -i 3 -c 2 var # a read() of 2 bytes
sysseek -u 3 0 # seek back to beginning
# or sysseek -u 3 -w current -2 # to seek back 2 bytes
syswrite -o 3 something-else
exec 3<&- # close

In ksh93:

exec 3<> "$file"
var=$(dd bs=2 count=1 <&3 2>/dev/null; echo .)
var=${var%?}
exec 3<#((0)) # seek to beginning
# or exec 3<#((CUR-2)) # to seek back 2 bytes
print -ru3 something-else

Portably, you could still open the file several times, for each offset you want, like here to read and write 2 bytes at offset 2 (provided they're not bytes with value 0 if not using zsh):

var=$(dd bs=2 count=1 skip=1 < "$file"; echo .)
var=${var%?}
printf %s something-else | dd bs=2 seek=1 1<> "$file"

Or:

printf %s something-else | dd bs=2 seek=1 of="$file" conv=notrunc

To read and write to the same file, ksh93 has two other interesting redirection operators:

tr 01 10 < file >; file

Would store the output of tr in a temporary file and if tr is successful, rename that to file (beware the file is created anew, so with possibly different permissions and ownership).

tr -d 0 < file 1<>; file

Same as the standard/Bourne tr -d 0 < file 1<> file except that if tr succeeds, file is truncated where tr finished writing. You can use that for filter commands that produce less output than they read input, or more precisely commands that would not read data that they've previously written.

And zsh has the =(...) form of process substitution which you can use as:

 mv =(tr 01 10 < file) file

(with similar effect and caveats as ksh93's >;). Or:

 cp =(tr 01 10 < file) file

which would preserve attributes of file but means an extra copy.

Now if you need to read and write at the same offset using the same file descriptor and neither zsh nor ksh93 are available, you could always revert to perl/python/ruby...

perl -e '
  open F, "<>", "file" or die "open: $!";
  read F, $var, 1;
  seek F, 0, 0;
  print F "something-else"'

Now, after re-reading the updated version of your question, it looks like your file is behaving more like a socket or bidirectional pipe, and not like a regular, seekable file.

In which case, it could be just a matter of:

socat - file:your-file

or:

(cat >&3 3>&- & cat <&3 3<&-) 3<> your-file

to feed data from and to that file as read from/to stdin/stdout.

Note that each cat reads/writes to its own copy of the file descriptor 3 open by the shell, but they share the same open file description so it should be equivalent.

Best Answer

Related Solutions

Command that prints file contents given filename on stdin

Bash – Read from and write to file from stdin and stdout

Related Question