Shell – Implement Circular Flow of Data Among Commands

command linepipescriptingshell

I know of two types how commands can be connected with each other:

by using a Pipe (putting std-output into std-input of the next
command).
by using a Tee (splice the output into many outputs).

I do not know if that is all that is possible,
so I draw a hypothetical connection type:

How could it be possible to implement a circular flow of data among commands like for instance in this pseudo code, where I use variables instead of commands.:

pseudo-code:

a = 1    # start condition 

repeat 
{
b = tripple(a)
c = sin(b) 
a = c + 1 
}

Best Answer

Circular I/O Loop Implemented with `tail -f`

This implements a circular I/O loop:

$ echo 1 >file
$ tail -f file | while read n; do echo $((n+1)); sleep 1; done | tee -a file
2
3
4
5
6
7
[..snip...]

This implements the circular input/output loop using the sine algorithm that you mentioned:

$ echo 1 >file
$ tail -f file | while read n; do echo "1+s(3*$n)" | bc -l; sleep 1; done | tee -a file
1.14112000805986722210
.72194624281527439351
1.82812473159858353270
.28347272185896349481
1.75155632167982146959
[..snip...]

Here, bc does the floating point math and s(...) is bc's notation for the sine function.

Implementation of the Same Algorithm Using a Variable Instead

For this particular math example, the circular I/O approach is not needed. One could simply update a variable:

$ n=1; while true; do n=$(echo "1+s(3*$n)" | bc -l); echo $n; sleep 1; done
1.14112000805986722210
.72194624281527439351
1.82812473159858353270
.28347272185896349481
[..snip...]

Related Solutions

shell – How to Send stdout to Multiple Commands

You can use tee and process substitution for this:

cat file.txt | tee >(pbcopy) | grep errors

This will send all the output of cat file.txt to pbcopy, and you'll only get the result of grep on your console.

You can put multiple processes in the tee part:

cat file.txt | tee >(pbcopy) >(do_stuff) >(do_more_stuff) | grep errors

Shell Pipe Tee – How to Reuse Pipe Data for Different Commands

You may use tee to duplicate command for processing whole stream by many command:

( ( seq 1 10 | tee /dev/fd/5 | sed s/^/line..\ / >&4 ) 5>&1 | wc -l ) 4>&1 
line.. 1
line.. 2
line.. 3
line.. 4
line.. 5
line.. 6
line.. 7
line.. 8
line.. 9
line.. 10
10

or split line by line, using bash:

while read line ;do
    echo cmd1 $line
    read line && echo cmd2 $line
    read line && echo cmd3 $line
  done < <(seq 1 10)
cmd1 1
cmd2 2
cmd3 3
cmd1 4
cmd2 5
cmd3 6
cmd1 7
cmd2 8
cmd3 9
cmd1 10

Finaly there is a way for running cmd1, cmd2 and cmd3 only once with 1/3 of stream as STDIN:

( ( ( seq 1 10 |
         tee /dev/fd/5 /dev/fd/6 |
           sed -ne '1{:a;p;N;N;N;s/^.*\n//;ta;}' |
           cmd1 >&4
     ) 5>&1 |
       sed -ne '2{:a;p;N;N;N;s/^.*\n//;ta;}' |
       cmd2 >&4
  ) 6>&1 |
    sed -ne '3{:a;p;N;N;N;s/^.*\n//;ta;}' |
    cmd3 >&4
) 4>&1 
command_1: 1
command_1: 4
command_1: 7
command_1: 10
Command-2: 2
Command-2: 5
Command-2: 8
command 3: 3
command 3: 6
command 3: 9

For trying this, you could use:

alias cmd1='sed -e "s/^/command_1: /"' \
    cmd2='sed -e "s/^/Command_2: /"' \
    cmd3='sed -e "s/^/Command_3: /"'

For using one stream on different process if on same script, you could do:

(
    for ((i=(RANDOM&7);i--;));do
        read line;
        echo CMD1 $line
      done
    for ((i=RANDOM&7;i--;));do
        read line
        echo CMD2 $line
      done
    while read line ;do
        echo CMD3 $line
      done
)
CMD1 1
CMD1 2
CMD1 3
CMD2 4
CMD2 5
CMD2 6
CMD2 7
CMD2 8
CMD2 9
CMD3 10

For this, you may have to transform your separated scripts into bash function to be able to build one overall script.

Another way could be to ensure each script won't output anything to STDOUT, than add a cat at end of each script to be able to chain them:

#!/bin/sh

for ((i=1;1<n;i++));do
   read line
   pRoCeSS the $line
   echo >output_log
 done

cat

Final command could look like:

seq 1 10 | cmd1 | cmd2 | cmd2