Shell – How to process a while-do loop and sort the iterated output

shell-scriptsort

I'm very new at Bash Scripting so forgive my ignorance. I have a text file with last names that I am reading and processing line-by-line an average of three test scores. I need to output the lines sorted by last name. Here is what I have to compute the average, with a variable named OUTTOSORT that I was trying to pass to sort before being output.

#!/bin/bash
FILE=$1
OUTTOSORT=""

while read -r SID FIRST LAST S1 S2 S3
do
    SUM=$(expr $S1 + $S2 + $S3)
    AVG=$(expr $SUM / 3)
    OUTTOSORT = OUTTOSORT + "$AVG [$SID] $LAST, $FIRST \n"
done < "$FILE"

sort < $OUTTOSORT

When run:

engr2-6-52-dhcp:homework6 Matt$ ./Grades.sh 
./Grades.sh: line 10: : No such file or directory
./Grades.sh: line 12: $OUTTOSORT: ambiguous redirect

Any suggestions on how to sort these lines after computing the average would be very helpful. Thanks!

Edit:

Thanks very much for your answers! I now have a problem where it's not sorting by the field I specify however:

#!/bin/bash
FILE=$1
COUNT=0
while read -r SID FIRST LAST S1 S2 S3 
do
    SUM=$(expr $S1 + $S2 + $S3)
    AVG=$(expr $SUM / 3)
    printf '%d [%d] %s, %s (%d)\n' "$AVG" "$SID" "$LAST" "$FIRST" "$COUNT"
    COUNT=$((COUNT + 1))
done < "$FILE" | sort -k 3,3n -k 4,4n -k 2,2g

Output:

71 [299226663] Camp, Laney (5)
80 [434401929] Camp, Skyler (6)
81 [199144454] Camp, Tracey (4)
82 [123456789] Johnson, Lee (0)
82 [928441032] Forester, Jess (7)
91 [999999999] Smith, Jaime (1)
92 [888111818] Forney, JC (2)
93 [928441032] Forester, Chris (8)
99 [290010111] Lee, Terry (3)

As you can see, it is sorting by average score instead of by Last name, First name, SID. I added a counter to confirm it was sorting across the lines of the output. What error is causing the sort to not use the correct keys? Thanks!

Edit:

Solution to sort problem found: type flags for last name and first name should be d instead of n, making the sort command the while loop command pipes to sort -k 3,3d -k 4,4d -k 2,2g.

Best Answer

You can print the results to standard output, then pipe them to the sort command:

while read -r SID FIRST LAST S1 S2 S3
do
     SUM=$(expr $S1 + $S2 + $S3)
     AVG=$(expr $SUM / 3)
     printf '%d [%d] %s, %s\n' "$AVG" "$SID" "$LAST" "$FIRST"
done < "$FILE" | sort

Related Solutions

Shell Script – Adding Lines to the Beginning and End of a Huge File

sed -i uses tempfiles as an implementation detail, which is what you are experiencing; however, prepending data to the beginning of a data stream without overwriting the existing contents requires rewriting the file, there's no way to get around that, even when avoiding sed -i.

If rewriting the file is not an option, you might consider manipulating it when it is read, for example:

{ echo some prepended text ; cat file ; } | command

Also, sed is for editing streams -- a file is not a stream. Use a program that is meant for this purpose, like ed or ex. The -i option to sed is not only not portable, it will also break any symlinks to your file, since it essentially deletes it and recreates it, which is pointless.

You can do this in a single command with ed like so:

ed -s file << 'EOF'
0a
prepend these lines
to the beginning
.
$a
append these lines
to the end
.
w
EOF

Note that depending on your implementation of ed, it may use a paging file, requiring you to have at least that much space available.

Sum Columns with awk – How to Sum Pair of Columns Based on Matching Fields

I would do this in Perl:

$ perl -lane '$k{"$F[0] $F[1]"}+=$F[2]+$F[3]; 
              END{print "$_ $k{$_}" for keys(%k) }' file 
2 1019 15
2 1021 4
2 1030 6
2 1031 8
2 1022 9

Or awk:

awk '{a[$1" "$2]+=$3+$4}END{for (i in a){print i,a[i]}}' file

If you want the output sorted according to the second column you could just pipe to sort:

awk '{a[$1" "$2]+=$3+$4}END{for (i in a){print i,a[i]}}' file | sort -k2

Note that both solutions include the 1st column as well. The idea is to use the first and second columns as keys to a hash (in perl) or an associative array (in awk). The key in each solution is column1 column2 so if two lines have the same column two but a different column one, they will be grouped separately:

$ cat file
2 1019 2 3
2 1019 4 1
3 1019 2 2

$ awk '{a[$1" "$2]+=$3+$4}END{for (i in a){print i,a[i]}}' file
3 1019 4
2 1019 10

Best Answer

Related Solutions

Shell Script – Adding Lines to the Beginning and End of a Huge File

Sum Columns with awk – How to Sum Pair of Columns Based on Matching Fields

Related Question