Files – Compare Two Folders for Missing Files

filenamesfiles

I have two different folders each with the same file names in it but one has missing files. How am I able to compare the two folders Folder1 and Folder2 and list the files missing in Folder2 which Folder1 contains.

Best Answer

$ tree
.
|-- dir1
|   |-- file1
|   |-- file2
|   |-- file3
|   |-- file4
|   `-- file5
`-- dir2
    |-- file2
    |-- file4
    `-- file5

2 directories, 8 files

$ for f1 in dir1/*; do f2="dir2/${f1#dir1/}"; [ ! -e "$f2" ] && printf '%s\n' "$f2"; done
dir2/file1
dir2/file3

This loops through all the names in the first directory, and for each creates the corresponding name of a file expected to exist in the second directory. If that file does not exist, its name is printed.

The loop, written out more verbosely (and using basename rather than a parameter substitution to delete the directory name from the pathname of the files in the first directory):

for f1 in dir1/*; do
    f2="dir2/$( basename "$f1" )"
    if [ ! -e "$f2" ]; then
        printf '%s\n' "$f2"
    fi
done

If the files in the two directories not only have the same names, but also the same contents, you may use diff (note: BSD diff used here, GNU diff may possibly say something else):

$ diff dir1 dir2
Only in dir1: file1
Only in dir1: file3

If the file contents of files with identical names differ, then this would obviously output quite a lot of additional data that may not be of interest. diff -q may quiet it down a bit in that case.

See also the diff manual on your system.

For comparing deeper hierarchies, you may want to use rsync:

$ rsync -r --ignore-existing -i -n dir1/ dir2
>f+++++++++ file1
>f+++++++++ file3

The above will output a line for each file anywhere under dir1 that does not have a corresponding file under dir2. The -n option (--dry-run) makes sure that no file is actually transferred to dir2.

The -r option (--recursive) makes the operation recursive and -i (--itemize-changes) selects the particular output format (the >f and the pluses indicates that the file is a new file on the receiving end).

Related Solutions

Rsync – How to Compare Two Folders and Copy Differences

I am not sure whether you can do it with any existing linux commands such as rsync or diff. But in my case I had to write my own script using Python, as python has the "filecmp" module for file comparison. I have posted the whole script and usage in my personal site - http://linuxfreelancer.com/

It usage is simple - give it the absolute path of new directory, old directory and difference directory in that order.

#!/usr/bin/env python

import os, sys
import filecmp
import re
from distutils import dir_util
import shutil

holderlist = []


def compareme(dir1, dir2):
    dircomp = filecmp.dircmp(dir1, dir2)
    only_in_one = dircomp.left_only
    diff_in_one = dircomp.diff_files
    dirpath = os.path.abspath(dir1)
    [holderlist.append(os.path.abspath(os.path.join(dir1, x))) for x in only_in_one]
    [holderlist.append(os.path.abspath(os.path.join(dir1, x))) for x in diff_in_one]
    if len(dircomp.common_dirs) > 0:
        for item in dircomp.common_dirs:
            compareme(
                os.path.abspath(os.path.join(dir1, item)),
                os.path.abspath(os.path.join(dir2, item)),
            )
        return holderlist


def main():
    if len(sys.argv) > 3:
        dir1 = sys.argv[1]
        dir2 = sys.argv[2]
        dir3 = sys.argv[3]
    else:
        print "Usage: ", sys.argv[0], "currentdir olddir difference"
        sys.exit(1)

    if not dir3.endswith("/"):
        dir3 = dir3 + "/"

    source_files = compareme(dir1, dir2)
    dir1 = os.path.abspath(dir1)
    dir3 = os.path.abspath(dir3)
    destination_files = []
    new_dirs_create = []
    for item in source_files:
        destination_files.append(re.sub(dir1, dir3, item))
    for item in destination_files:
        new_dirs_create.append(os.path.split(item)[0])
    for mydir in set(new_dirs_create):
        if not os.path.exists(mydir):
            os.makedirs(mydir)
    # copy pair
    copy_pair = zip(source_files, destination_files)
    for item in copy_pair:
        if os.path.isfile(item[0]):
            shutil.copyfile(item[0], item[1])


if __name__ == "__main__":
    main()

Merge two image folders

Since you are new to the command line I will go over the steps one by one, explaining each along the way.

When you open up a terminal/shell/command-line, you'll see a command-prompt like so:

username:~ $

or something similar. You'll almost certainly start out in your home directory, which has the name of your username.

Let's say your two image folders are folder1 and folder2. You said they are on the Desktop, so first we change to the Desktop directory:

cd Desktop

The cd command is the "change directory" command, doing exactly what it sounds like.

Now, we are going to use the mv command, which of course stands for "move".

mv -i folder1/* folder2/

I used the -i flag, which stands for "interactive", which causes a prompt to appear every time there is a file-name conflict. You'll be asked whether you want to overwrite the destination file or not.

After this, you'll have to manually resolve any conflicts* by renaming the files left in folder1 so they don't conflict with the files in folder2. Finally, run the above command again, but this time you should get no conflicts.

After all this, your folder1 directory should be empty, so go ahead and remove it with

rmdir folder1

and you're done!

I really recommend you go through a simple bash tutorial like this one to learn the basics.

Note that it is of course possible to resolve naming conflicts programmatically by renaming each of the files in folder1, but I tried to keep things simple.

Best Answer

Related Solutions

Rsync – How to Compare Two Folders and Copy Differences

Merge two image folders

Related Question