Compare two folder structures and list files which exist in both, but are different

finderterminal

I have a local working folder which mirrors part of a web server’s public folder. I usually work in the local copy and then auto-upload files to the server when saving. Trouble is, I’ve realised lately that a lot of files in my local files appear to be out of date, so whenever I save and upload a file, I’m potentially overwriting a newer version. This is obviously problematic, so I’d like to get all the out-of-date local files up to date.

The best way I can think of to do this is to download the entire public folder as it is and compare each file with my local copy, going manually through files with differences (by comparing them in Visual Studio Code). The public server folder has about 5GB of extra stuff that I don’t need (or want) in my local working folder, though, so I’d have to filter out the unwanted stuff first.

In other words, I’m looking for a way (GUI or terminal) to do the following:

Give two top-level directories as input
Iterate recursively through both directories and pick out files which exist in both (in the same relative location)
Compare each set of matched files and list the ones where the two files are not identical

Is there some reasonably non-complex way of doing this?

Best Answer

While I'm a big proponent of utilizing built in tools (+1 for nohillside's answer) rather than searching for the nearest app, from time to time I do find really invaluable applications. In this case, I found a graphical differencing and merging application called Meld.

Meld is a visual diff and merge tool targeted at developers. Meld helps you compare files, directories, and version controlled projects. It provides two- and three-way comparison of both files and directories, and has support for many popular version control systems.

Meld is probably overkill for what you're looking to do, but not so much that it makes the software cumbersome to use. The best part is that it's free (as in beer).

Related Solutions

Use curl to download images from website using wildcard

You can use this bash code for the actual URL you provided in your comment.

for i in $(for j in {1..8}; do
    curl http://occipital.com/user/01ca-255/george-coghill/$j/;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  curl -L -o "${i##*/}" "$i";
done

Feel free to write this command as one line; all line breaks were only added to increase legibility. You may copy or remove them, whatever you like.

What this code does:

Iterate over the 8 pages of your account gallery
Extract the image names from the preview images
Fetch all full-sized images using this list of names

If you want to only download files which don't exist, and don't know the number of gallery pages up front, you can adjust the code to this:

for i in $(
  j=1;
  while curl --max-redirs 0 -L -f \
      http://occipital.com/user/01ca-255/george-coghill/$((j++))/; do
    :;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  [[ -f "${i##*/}" ]] || curl -L -o "${i##*/}" "$i";
done

The first code now increments the page number until a redirect or error occurs. The server will redirect you to the last existing page if you try to visit a page after the existing ones. The [[ -f … ]] || part will execute curl only if the corresponding file does not exist yet.

MacOS – Files Greyed out in Finder – Potential extended attributes issue

This worked, but after external disk (NTFS) eject the problem was back

xattr -d com.apple.FinderInfo fileToUpdate

(or for all files in a folder xattr -d com.apple.FinderInfo *)

This worked after eject/re-mount

xattr -wx com.apple.FinderInfo 

0000000000000000001000000000000000000000000000000000000000000000 fileToUpdate

Notes from man xattr:

 xattr -w [-rsx] attr_name attr_value file ...
 xattr -x  Force the attribute value to be displayed in the hexadecimal representation.

Best Answer

Related Solutions

Use curl to download images from website using wildcard

MacOS – Files Greyed out in Finder – Potential extended attributes issue

Related Question