Bash – How to copy all files in a folder excluding files which are being written

bashfile-copyfilesfindshell-script

I download multiple files to a folder downloading via HTTPie. A bash script aims to process downloaded files, and I tried to copy the downloaded files to another folder as

find /folder/downloading -type f -exec mv '{}' /folder/downloaded \;

but this also copies the files, which have not been finished yet. I tried to limit the transfer to older files by adding -mmin +5 to the command. What is the efficient command to leave the files are being written and transfer downloaded files only?

Best Answer

Not very efficient, but you could do:

find /folder/downloading -type f -exec sh -c '
  for file do
    lsof -F a "$file" | grep -q w || mv "$file" /folder/downloaded
  done' sh {} +

That is check that the file is not listed with a write access mode in the list of open files before moving.

The psmisc implementation of fuser as typically found on Linux-based operating systems has a -w function (to check for files open for writing) but unfortunately it only work with -k to kill the corresponding processes. However, it seems you can still use it by using the pseudo-signal 0 that does nothing:

find /folder/downloading -type f -exec sh -c '
  for file do
    fuser -s -w -k -0 "$file"  || mv "$file" /folder/downloaded
  done' sh {} +

Remove the -s (or even replace it with -v) if you want to see what process(es) is(are) preventing the move.

Note that if you're not running those commands as super-user, you will only get information about your processes. If the processes downloading the files are running as a different user, they will remain undetected.

Also note that unless you're moving the files to a different file system, moving the files will not prevent whatever process is currently writing to the file from finishing writing to it.

However, depending of what they've been designed to do afterwards, they might be confused if after they finish writing, the file is no longer there (for instance if they want to change some attributes of the file after downloading it and do so not via the file descriptor (like chmod() vs fchmod(), or utimes() which cannot be done via a file descriptor)).

Related Solutions

Rsync – How to Compare Two Folders and Copy Differences

I am not sure whether you can do it with any existing linux commands such as rsync or diff. But in my case I had to write my own script using Python, as python has the "filecmp" module for file comparison. I have posted the whole script and usage in my personal site - http://linuxfreelancer.com/

It usage is simple - give it the absolute path of new directory, old directory and difference directory in that order.

#!/usr/bin/env python

import os, sys
import filecmp
import re
from distutils import dir_util
import shutil

holderlist = []


def compareme(dir1, dir2):
    dircomp = filecmp.dircmp(dir1, dir2)
    only_in_one = dircomp.left_only
    diff_in_one = dircomp.diff_files
    dirpath = os.path.abspath(dir1)
    [holderlist.append(os.path.abspath(os.path.join(dir1, x))) for x in only_in_one]
    [holderlist.append(os.path.abspath(os.path.join(dir1, x))) for x in diff_in_one]
    if len(dircomp.common_dirs) > 0:
        for item in dircomp.common_dirs:
            compareme(
                os.path.abspath(os.path.join(dir1, item)),
                os.path.abspath(os.path.join(dir2, item)),
            )
        return holderlist


def main():
    if len(sys.argv) > 3:
        dir1 = sys.argv[1]
        dir2 = sys.argv[2]
        dir3 = sys.argv[3]
    else:
        print "Usage: ", sys.argv[0], "currentdir olddir difference"
        sys.exit(1)

    if not dir3.endswith("/"):
        dir3 = dir3 + "/"

    source_files = compareme(dir1, dir2)
    dir1 = os.path.abspath(dir1)
    dir3 = os.path.abspath(dir3)
    destination_files = []
    new_dirs_create = []
    for item in source_files:
        destination_files.append(re.sub(dir1, dir3, item))
    for item in destination_files:
        new_dirs_create.append(os.path.split(item)[0])
    for mydir in set(new_dirs_create):
        if not os.path.exists(mydir):
            os.makedirs(mydir)
    # copy pair
    copy_pair = zip(source_files, destination_files)
    for item in copy_pair:
        if os.path.isfile(item[0]):
            shutil.copyfile(item[0], item[1])


if __name__ == "__main__":
    main()

Find 2 files with similar names and move them to a new location

The simplest approach will depend on how much you can trust your users. If you don't need to test that both files exist or that the names are correct or anything, you don't even need a script. You can do this with a simple find:

find /dir/to/source -name "ABC*" -mmin +5 -exec mv {} /destination/dir \;

If you need to make sure that i) both files are there and ii) both have a modification time of at least 5 minutes ago, on a GNU system, you could do this:

#!/usr/bin/env bash

SOURCEDIR="/dir/to/source"
DESTDIR="/destination/dir"

for f in "${SOURCEDIR}"/*.xml; do
    ## Make sure the file exists and is a regular file (or symlink to regular file),
    ## and that its modification date is at least 5 minutes ago
    [ -f "$f" ] && [ "$(( $(date +%s) - $(stat -c %Y "$f") ))" -ge 300 ] || continue

    ## Do the same for a file of the same name but with the .mxf extension.
    mxf="${SOURCEDIR}/$(basename "$f" .xml).mxf";
    [ -f "$mxf" ] && [ "$(( $(date +%s) - $(stat -c %Y "$no_ext") ))" -ge 300 ] || continue

    ## We will only get to this point if all of the above tests were successful
    echo mv -v "$f" "$mxf" "$DESTDIR"
done

Best Answer

Related Solutions

Rsync – How to Compare Two Folders and Copy Differences

Find 2 files with similar names and move them to a new location

Related Question