Binary diff of two folders

Git 2.9 was released earlier this year which included the experimental flag --compaction-heuristic on the git diff command:

In 2.9, Git's diff engine learned a new heuristic: it tries to keep hunk boundaries at blank lines, shifting the hunk "up" whenever the bottom of the hunk matches the bottom of the preceding context, until we hit a blank line.

I don't think GitHub has it enabled for diffs on the web UI for Pull Requests and comparisons, but you can do it locally. I'd recommend using it in conjunction with --word-diff if you need that level of granularity.

More details available on the GitHub blog: https://github.com/blog/2188-git-2-9-has-been-released

One approach could be to compute the Levenshtein distance.

Here using the Text::LevenshteinXS perl module:

distance() {
  perl -MText::LevenshteinXS  -le 'print distance(@ARGV)' "$@"
}

Then:

$ distance foo foo
0
$ distance black blink
2
$ distance "$(cat /etc/passwd)" "$(tr a b < /etc/passwd)"
177

Here's a line-based implementation of the Levenshtein distance in awk (computes the distance in terms of number of inserted/deleted/modified lines instead of characters):

awk '
  {if (NR==FNR) s[++m]=$0; else t[++n]=$0}
  function min(x, y) {
    return x < y ? x : y
  }
  END {
    for(i=0;i<=m;i++) d[i,0] = i
    for(j=0;j<=n;j++) d[0,j] = j

    for(i=1;i<=m;i++) {
      for(j=1;j<=n;j++) {
        c = s[i] != t[j]
        d[i,j] = min(d[i-1,j]+1,min(d[i,j-1]+1,d[i-1,j-1]+c))
      }
    }
    print d[m,n]
  }' file1 file2

You may also be interested in diffstat's output:

$ diff -u /etc/passwd <(tr a b < /etc/passwd) | diffstat
 13 |  114 ++++++++++++++++++++++++++++++++++-----------------------------------
 1 file changed, 57 insertions(+), 57 deletions(-)