Bash – How to print lines with blank 5th field in CSV

With sed:

sed '$!N;/\(.*,\).*\n\1/!P;D' infile

N means there are always two consecutive lines in the pattern space and sed Prints the first one of them only if the first field in that line is not the same as the first field in the second line. Then D removes the first line from the pattern space and restarts the cycle.

Another way with gnu datamash (assuming your file is sorted as datamash requires sorted input):

datamash -t ',' -g 1 last 2 <infile

This groups the , delimited input by 1st field, printing only the last value (from 2nd column) of each group.

If your file isn't sorted datamash can sort it via -s:

datamash -t ',' -s -g 1 last 2 <infile

but that means the initial order of lines will not be preserved. So this might not do what you want. In that case you could use sed/awk/perl etc...

$ awk -F, 'NR==FNR{a[$1,$2]++; next} a[$1,$2]>1' file.txt file.txt 
spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}
spark2-thrift-sparkconf,spark.history.fs.logDirectory,true

Two file processing using same input file twice

• NR==FNR{a[$1,$2]++; next} using first two fields as key, save number of occurrences
• a[$1,$2]>1 print only if count is greater than 1 during second pass

For the opposite case, simple matter of changing condition check

$ awk -F, 'NR==FNR{a[$1,$2]++; next} a[$1,$2]==1' file.txt file.txt 
spark2-thrift-sparkconf,spark.history.Log.logDirectory,true
spark2-thrift-sparkconf,spark.history.DF.logDirectory,true