Bash – Optimizing a shell script with long running while loop

Oh my!

I can see why it takes forever to run, you're repeating operations, not caching information and pretty much beating the computer to death. Poor Computer. :(

Awk is not a light-weight, and you're invoking it many, many times over the same data. I was able to run it once and set all five variables.

Without knowing what this is supposed to be doing or accomplishing, there's just so much that can be done.

Considering that ALL of the processing is grep's, awk's, sed's and tr's, you could get an impressive speed boost by writing this script in PERL. PERL is/was designed to handle text and reports. It can do all those grep/awk/sed/tr internally without shelling out to another program repeatedly.

But here's some improvements:

if [ -s "$LOC/check.txt" ]; then

function setvars() {
    CONN_TSMP="$1"
    USER="$2"
    HOST="$3"
    DB="$4"
    IP="$5"
    return
}
    while read line; do
        echo " started processing ${line} at $(date) " >> "${SCRIPT_LOC}/running_status.txt"
        ID=$(echo "$line" | tr -d '"')
        # are you sure you don't want the FIRST match?  This will give ALL the matches,
        # which will prevent you from getting good values for the variables
        # to only get first entry that matches:
        # TST=$(grep --max-count=1 -w "$line" "$PERM_LOC/id_processing.txt")
        # (or -m 1, but long options document what you're doing better)
        TST=$(grep -w "$line" "$PERM_LOC/id_processing.txt")
        VARS=$(echo "${TST}" | awk -F '"' '{print "\""$2"\" \""$10"\" \""$18"\" \""$20"\" \""$22'})
        #                                        CONN_TSMP     USER      HOST      IP        DB
        # magic!  setvars receives the 5 values awk pulled out (ran it once!)
        # NO QUOTES on next line, already has them embedded from awk
        setvars $VARS

        if [ -z "$IP" ]; then
            IP="$HOST"
        fi

        CMD_TARGET="/data1/commands/user_commands"
        FILE="${ID}-${CONN_TSMP}-${USER}@${IP}.txt"

        if [ "$USER" == "root" ] && [ -z "$DB" ]; then
            TARGET="/data1/sessions/root_sec"
        else
            TARGET="/data1/sessions/user_sec"
        fi

        # does this need to be redirected to a file?
        ls "$TARGET/$FILE"
        if [ $? -ne 0 ]; then
            # awk can likely do the print and the removal of </> characters in
            # one pass (my awk-fu is weak this morning)
            echo "$TST" | awk -F 'STATUS="0"' '{print $2}'| sed "s/[</>]//g" >> "$TARGET/$FILE"
            echo -e "\n" >> "$TARGET/$FILE"
        fi

        # ALWAYS quote your values, embedded spaces will bite you!
        grep "$line" "$LOC/out.txt" > "$LOC/temp.txt"

        while read val; do
            TSMP=$(echo "$val" | awk -F '"' '{print $2}')
            QUERY=$(echo "$val" | awk -F 'SQLTEXT=' '{print $2}' | sed "s/[\"/]//g")
            echo " TIMESTAMP=$TSMP " >> "$TARGET/$FILE"
            echo " QUERY=$QUERY " >> "$TARGET/$FILE"
            TEXT=$(grep "$QUERY" "$PERM_LOC/commands.txt")
            if [ -n "$TEXT" ]; then
                NUM=$(expr $NUM + 1)
                # could also be:  NUM=$(($NUM+1)) (bash v4.0+)
                SUB_FILE="$ID-$command-$NUM-$TSMP-$USER@$IP.txt"
                echo -e "===============\n" > "$CMD_TARGET/$SUB_FILE"
                echo "FILE      =   \"$SUB_FILE\"" >> "$CMD_TARGET/$SUB_FILE"
                ### same way append 6 more lines to $SUB_FILE

                SUB=$(echo "$WARN_ME" | grep "$command")
                if [ "$command" == "$VC" ]; then
                    STATE=" very critical "
                elif [ -z "$SUB" ]; then
                    STATE=" CRITICAL "
                else
                    STATE=" WARNING "
                fi

                if [ "$USER" != "root" -a "$command" != "$VC" ]; then
                    # this should probably be $command instead of command?
                    # oh wait, probably a placeholder statement
                    mail command &
                elif [ "$USER" == "root" -a -z "$HOST" ]; then
                    mail command &
                elif [ "$USER" == "root" -a "$command" == "$VC" ]; then
                    mail command &
                else
                    echo -e "some message \n" >> $LOC/operations.txt
                fi
            fi
        done < "$LOC/temp.txt"
    done < "$LOC/check.txt"
fi

Hmm, "shell script only". Well, with that in mind, perhaps you could pre-grep "$LOC/check.txt" and/or "$LOC/temp.txt" so that you could use the 'already grepped' output instead of grepping in the loop.

The more I look at it, the more convinced I am that awk could likely do all this work in a single pass through the data... AND process EVERY entry, not just the first one (as I pointed out in the comments, you really need another loop between the "read line" and "read var" loops.)

It'd be a long awk script, but definitely doable. And awk is worth knowing, take a moment and play with it, it's not that difficult, just different. Grok Awk!