Shell – How to skip first, last non-blank line and blank lines from modification in a file

awksedshell-scripttext processing

I have a file something like this:

H|ACCT|XEC|1|TEMP|20130215035845

849002|48|1208004|100|||1

849007|28|1208004|100|||1

T|2|3

Note that there are extra empty lines at the end of the file.

I want to replace the value of column 5 with column 4's value in all the lines except first and last non-blank line.

I cannot rely on the number of fields as the last line may have as many fields as the other ones, nor on the lines to modify always starting with a number.

I tried the code below:

awk 'BEGIN{FS="|"; OFS="|"} {$5=$4; print}' in.txt

Output is:

H|ACCT|XEC|1|1|20130215035845
||||
849002|48|1208004|100|100||1
||||
849007|28|1208004|100|100||1
||||
T|2|3||
||||
||||
||||

Expected output:

H|ACCT|XEC|1|TEMP|20130215035845|

849002|48|1208004|100|100||1

849007|28|1208004|100|100||1

T|2|3

How can I skip the first and last non-blank lines from getting changed? I also want to skip blank lines.

Best Answer

Here you go with awk and processing the file only once.

awk -F'|' 'NR==1{print;next} m && NF{print m}
    NF{l="\n"$0; $5=$4; m="\n"$0; c=0}; !NF{c++}
END{ print l; for (; i++<c;)print }' OFS='|' infile

Explanation:

Here we are skyping first line to being replace 5^th field's value with 4^th field's value, and just print it and do next.

... if it (current next line) was not empty line (at least contains one field NF), then take a backup of whole line with a \newline added l="\n"$0 first next set 5^th field's value with 4^th field's value $5=$4 and last set it to a variable m with a \newline added m="\n"$0;; There is a c variable as a counter flag and is used to determine the number of empty lines !NF{c++} if no line with at least one field seen; Otherwise c=0 will reset this counter.

Now we have modified line in m variable and m && NF{print m} will print it where in the next step awk runs and m has set and it's not on empty lines & NF (this is used to prevent duplication on printing when empty line).

At the end we are printing the untouched last line which we take backup every time before performing replacement END{ print l; ... and then number of empty lines which never seen a line with a field with looping for (; i++<c;)print }'.

That's much shorter if you don't need redundant empty lines.

awk -F'|' 'NR==1{print;next} m && NF{print m}
    NF{l=$0; $5=$4; m=$0} END{ print l}' OFS='|' infile

Related Solutions

How to replace text in the last non-blank line

Answer

perl -0777 -p -i -e 's/,(\n*)\Z/\1/m' *.txt

will remove the last ',' in all files ending in .txt, if the ',' is followed only by 0-or-more newline characters then the end of the file.

From your example:

reedm@www:~/tmp $ cat > test.txt
blah blah blah,
blah blah blah,
blah blah blah,


reedm@www:~/tmp $ perl -0777 -p -i -e 's/,(\n*)\Z/\1/m' *.txt
reedm@www:~/tmp $ cat test.txt
blah blah blah,
blah blah blah,
blah blah blah


reedm@www:~/tmp $

Wat?

Perl is an esoteric beast at the best of times, and perl one-liners can be particularly cryptic.

The -e flag allows us to pass a perl program on the command line. In this case, the 's/regex/replace/flags' is the program.

The -p flag causes perl to apply your supplied program in a loop over each "line" (see -0) for each filename provided.

The -i flag causes perl to replace the file with the output of the program, rather than printing the output to standard out.

The -0 flag changes what delimiter perl uses to break a file into "lines". 0777 is a special value, used by convention to make perl read the entire file into a single "line".

The regular expression is somewhat complicated by the use of a few perl-specific tricks:

First, the m flag at the end causes the regex to operate on multiple lines.
, is simple, and matches a single, literal comma.
(\n*) matches 0-or-more newlines in a row, and stores them as a subpattern (the ( and ) characters denote a subpattern). As this is the first subpattern, we can use \1 in the replacement section to mean "whatever this subpattern matched".
\Z is a perl specific extension, and matches the end of the string being worked with -- in this case, that's the entire file.
In the replacement part, we use \1 to replace the match with only the series of newlines, removing the comma.

For man information on perl regular expressions and perl command line flags, check out the man pages for perlre and perlrun respectively.

How to remove multiple blank lines from a file

Case 1:

awk '!NF {if (++n <= 2) print; next}; {n=0;print}'

Case 2:

awk '!NF {s = s $0 "\n"; n++; next}
     {if (n>1) printf "%s", s; n=0; s=""; print}
     END {if (n>1) printf "%s", s}'

Best Answer

Related Solutions

How to replace text in the last non-blank line

How to remove multiple blank lines from a file

Related Question