Shell – How to include program which only works with in/out files into a pipeline

filespipeshell

I need to process image with several tools in a pipeline tool1 | tool2 | tool3 | .... It appeared though that one of the tools isn't designed to work in a pipeline and only works in format of user@computer:/~# bad_tool infile.png outfile.png.

Is there any way to include it into pipeline? I really want to avoid creating files for this only program and then removing them, etc.

Best Answer

if this satisfies you, here is proposition how to do this using pipes.

Assumption is that input and output files of "badtool" can be pipes.

mkfifo IF
mkfifo OF

# one therminal
tool | tool2 |... tooln > IF

# second terminal
bad_tool IF OF

#third terminal
tooln+1 < OF | tool n+2 | tool n+3 ...

If you would like to create script you can wrap those parts into functions:

function A(){ ... }
function B(){ ... }
function C(){ ... }
# and run in background in parallel
A&
B&
C&

Continue with all your images (pipes IF and OF are "reusable") and after whole job delete them

rm IF OF

Related Solutions

Shell – pipe, { list; } only works with some programs

I did some investigating using strace and it appears to be due to the way the program on the left side of the pipeline does it's writing to the terminal. When the ls command is executed it writes all of the data in a single write(). This causes head to consume all of stdin.

On the other hand ps writes out data in batches, so only the first write() is consumed by head, and then it exists. Later calls to write() will go to the newly spawned grep process.

This means that it would not work if the process you are trying to grep for did not occur in the first write(), since grep does not get to see all of the data (it sees even less than just the data minus the first line).

Here is an example of trying to grep for pid 1 on my system:

$ ps -eF | { head -n2; }
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root         1     0  0  1697  3768   2 Oct03 ?        00:00:03 /lib/systemd/systemd
$ ps -eF | grep '/lib/systemd/systemd$'
root         1     0  0  1697  3768   2 Oct03 ?        00:00:03 /lib/systemd/systemd
$ ps -eF | { head -n1; grep '/lib/systemd/systemd$'; }
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD

Your ps -eF example only works by chance.

Bash – Getting the process-id out of command launched with “su -c”

You can do:

su - appuser -c 'nohup java app.Main > /dev/null 2>&1 &
                 echo "$!"' > /var/run/app.pid

(that assumes the login shell of appuser is Bourne-like).

su - resets the environment, so if you want to have variables expanded in the command line, it has to be done by your shell (not the login shell of the remote user) like:

su - appuser -c "nohup '$JAVA_BIN' '$JAVA_CLASS' > /dev/null 2>&1 &"'
                 echo "$!"' > /var/run/app.pid

(that assumes those variables don't contain single quote characters).

You want the redirection to be performed by your shell (running as root), not appuser's shell (which probably doesn't have write permission to /var/run).

Best Answer

Related Solutions

Shell – pipe, { list; } only works with some programs

Bash – Getting the process-id out of command launched with “su -c”

Related Question