Text Processing – Remove Extra Header Lines from File Except for the First Line

If your fields can never contain whitespace, an empty field means either a tab as a first character (^\t), a tab as the last character (\t$) or two consecutive tabs (\t\t). You could therefore filter out lines containing any of those:

grep -Ev $'^\t|\t\t|\t$' file

If you can have whitespace, things get more complex. If your fields can begin with spaces, use this instead (it considers a field with only spaces to be empty):

grep -Pv '\t\s*(\t|$)|\t$|^\t' file

The change filters out lines matching a tab followed by 0 or more spaces and then either another tab or the end of the line.

That will also fail if the last field contains nothing but spaces. To avoid that too, use perl with the -F and -a options to split input into the @F array, telling it to print unless one of the fields is empty (/^$/):

perl -F'\t' -lane 'print unless grep{/^$/} @F' file

Using awk:

awk '/logical IO/ {if (!seen) {print; seen=1}; next}; {print; seen=0}' file.txt 

• /logical IO/ {if (!seen) {print; seen=1}; next} checks if the line contains logical IO, if found and the variable seen is false i.e. previous line does not contain logical IO, then print the line, set seen=1 and go to the next line else go to the next line as the previous line has logical IO

• For any other line, {print; seen=0}, prints the line and the sets seen=0

Example:

$ cat file.txt 
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
testing logical IO 500
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346
parsing logical IO 346
testing logical IO 12

$ awk '/logical IO/ {if (!seen) {print; seen=1}; next}; {print; seen=0}' file.txt 
select * from test1 where 1=1
testing logical IO 24
select * from test2 where condition=4
parsing logical IO 45
select * from test5 where 1=1
testing logical IO 24
select * from test5 where condition=78
parsing logical IO 346