Bash – How to Move Last 3 Lines of Pipeline to Top of Output with ed

awkbashedscriptingtee

I have a small AWK script that summarizes the output of another program, and I am trying to print that summary before the command itself, but I cannot figure out a good way to do it. My naive first solution was running the command twice, once to get the summary info and once to print the full output with the prepended summary. Now I am using tee to only read the program output once and using head and tail to move it around, which is about 3 times faster than the previous version.

The current version looks like

program | awk ... | tee >(head -n -3) >(tail -n -3) > /dev/null

which does give the desired output but is less elegant (and fast) than I'm sure is possible. At the very least I think there is a way not to have to redirect to /dev/null, but I can't figure out the arrangement that makes that happen.

I have seen some answers on similar questions using ed, and I would like to do this as well, but when I try piping my commands into ed it doesn't work. Specifically, I thought

echo -e "$-2,$m0\n,p\nQ" | ed <(program | awk ...)

should work since running those commands in ed itself does, but I have narrowed down the problem to the "$-2" address, which appears not to work when piping into ed but does work when running it directly. I've tested just that part on both GNU ed 1.9 and 1.16. Using the -v option in both cases just gives me the "Invalid command suffix" error. Even running

echo "$-2,$p" | ed -v filename

produces this error, so it's not a problem with the process substitution. I obviously doubt I've identified a bug in ed after using it for less than an hour, so I figured I would ask here to see what I'm doing wrong. I am also open to any solutions not involving ed, I just thought it seemed like a promising way to go.

If it helps, "program" is qstat and the AWK is just counting the number of jobs running, queued, and exiting for my user and printing the totals at the end. There are probably some qstat options that would do what I want built in, but I haven't found them in the manual yet so that would be a helpful answer as well! I would really like to understand what is going wrong with my ed command though.

Best Answer

You have actually stumbled on the shell, not on ed. This

echo -e "$-2,$m0\n,p\nQ"

means $- and $m0 undergo parameter expansion, as they are enclosed by double quotes. Run echo "$-" and echo "$m0" to see it for yourself. They should be enclosed in single quotes so that the shell does not expand them.

Since we are fixing it, let us also favor printf over echo. The latter has a non-uniform behavior accross implementations, while the former is sound. This should do:

printf '%s\n' '$-2,$m0' ',p' 'Q' | ed -s <(program | awk ...)

-s option has been added to ed, so as to "suppress diagnostics, byte counts and '!' prompt". This is purely cosmetic.

Sample execution (with a useless use of cat to simulate the process substitution):

$ cat input
ATOM    126  CD  GLN A 449      -2.853  11.592 119.709  1.00 17.95           C
ATOM    127  OE1 GLN A 449      -4.056  11.297 119.695  1.00 20.83           O
ATOM    128  NE2 GLN A 449      -1.948  10.876 120.359  1.00 14.98           N
HETATM  129  N   MSE A 450      -4.523  16.830 119.280  1.00 14.88           N
HETATM  130  CA  MSE A 450      -5.537  17.804 118.911  1.00 15.65           C

$ printf '%s\n' '$-2,$m0' ',p' 'Q' | ed -s <(cat input)
ATOM    128  NE2 GLN A 449      -1.948  10.876 120.359  1.00 14.98           N
HETATM  129  N   MSE A 450      -4.523  16.830 119.280  1.00 14.88           N
HETATM  130  CA  MSE A 450      -5.537  17.804 118.911  1.00 15.65           C
ATOM    126  CD  GLN A 449      -2.853  11.592 119.709  1.00 17.95           C
ATOM    127  OE1 GLN A 449      -4.056  11.297 119.695  1.00 20.83           O
Related Question