Shell – Efficient way of comparing in awk

awklinuxscriptingshell

#!/bin/awk
BEGIN {
        while(getline var < compareTo > 0)
        {
                orderIds[var]=var;
        }
}
{
        if(orderIds[$0] == "")
        {
                print $0;
        }
}

Running as 

awk -v compareTo="ids.log.remote" -f sample.awk ids.log.local

This is working, but instead of using associative arrays ( like HashMap ), is there anything like a HashSet in awk?

I got the timings

bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.130s
user    0m0.127s
sys     0m0.002s
bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.126s
user    0m0.125s
sys     0m0.000s
bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.131s
user    0m0.128s
sys     0m0.002s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.053s
user    0m0.051s
sys     0m0.003s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.052s
user    0m0.051s
sys     0m0.001s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.053s
user    0m0.051s
sys     0m0.002s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.066s
user    0m0.060s
sys     0m0.006s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.065s
user    0m0.058s
sys     0m0.008s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.061s
user    0m0.053s
sys     0m0.007s

@Dimitre Radoulov Looks like your awk is faster. Thanks.

Best Answer

I believe this is the most efficient way to do this in awk:

awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.remote ids.log.local

You may try with grep too:

grep -xFVf ids.log.remote ids.log.local
Related Question