Linux – Copy new files from a monitored folder to another in debian

debianfile-transferlinux

I have directory a and directory b. Directory a has new files and folders copied to it periodically. I would like to monitor folder a for those new files and automatically copy them to folder b. Unfortunately due to some organizational scripts that I have set up previously in directory b preclude me from using rsync for these purposes since the folder structure at the destination would most likely differ too greatly between rsync runs.

Is there any kind of alternative setup that I can use?

Best Answer

Another way of doing this would be to use inotify:

Install the inotify-tools package
```
 sudo apt-get install inotify-tools
```

Write a little script that uses inotifywatch to check your folder for changes and moves any new files to the target directory:

 #!/usr/bin/env bash

 ## The target and source can contain spaces as 
 ## long as they are quoted. 
 target="/path/to/target dir"
 source="/path to/source/dir";

 while true; do 

   ## Watch for new files, the grep will return true if a file has
   ## been copied, modified or created.
   inotifywatch -e modify -e create -e moved_to -t 1 "$source" 2>/dev/null |
      grep total && 

   ## The -u option to cp causes it to only copy files 
   ## that are newer in $source than in $target. Any files
   ## not present in $target will be copied.
   cp -vu "$source"/* "$target"/
 done

Save that script in your $PATH and make it executable, e.g.:
```
 chmod 744 /usr/bin/watch_dir.sh
```
Have it run every time your machine reboots, create a crontab (crontab -e, as described in @MariusMatutiae's answer) and add this line to it:
```
 @reboot /usr/bin/watch_dir.sh 
```

Now, every time you reboot, the directory will automatically be watched and new files copied from source to target.

Related Solutions

Linux – Recursive, Non-Overwriting File Copy

You could use rsync (it also does local copy)

rsync -r --ignore-existing --include=*/ --include=*.js --exclude=* source/ destination

-r to recurse into directories,
--ignore-existing to ignore existing files in destination,
the include and exclude filters mean: include all directories, include all *.js files, exclude the rest; the first include is needed, otherwise the final exclude will also exclude directories before their content is scanned.

Finally, you can add a -P if you want to watch progress, a --list-only if you want to see what it would copy without actually copying, and a -t if you want to preserve the timestamps.

This is not related, but I learned the rsync command recently, when I moved 15 years of documents from one partition to another. Confident that my files were there, I then wiped the old partition and put some other stuff in there; I realized later that I lost all the timestamps, and discovered the -t flag. Just wanted to share my distress :'(

Linux – rename all jpeg files sequentially in all nested subfolders and move to a new folder

With Larry Wall's rename/prename(*) you can easily "flatten" the structure:

cd mainfolder
rename -n 's:/:-:g' */*/*.jpg

This will move subfolder1/0001.jpg to subfolder1-0001.jpg (in mainfolder) because rename edits the whole input path and then does the equivalent of mv using source and edited names.

g in the edit means to replace all occurrences, useful if you have more than one level of subdirectories.
-n just shows what it would do without doing anything
remove or replace by -v for the actual action.
by default no overwrite can happen (you need a -f) but normally since this includes subdirectories in the names you shouldn't have name clashes anyway.
once everything is in mainfolder, it is easy to do a bulk move elsewhere if needed.
if the files are in various levels of nesting, use shopt -s globstar and use ** to denote all levels:
```
rename s:/:-:g **/*.jpg
```

For instance:

Starting with

.
├── sub1
│   ├── subsub1
│   │   ├── 001.jpg
│   │   ├── 002.jpg
│   │   └── 003.jpg
│   └── subsub2
│       ├── 001.jpg
│       ├── 002.jpg
│       └── 003.jpg
└── sub2
    ├── subsub1
    │   ├── 001.jpg
    │   ├── 002.jpg
    │   └── 003.jpg
    └── subsub2
        ├── 001.jpg
        ├── 002.jpg
        └── 003.jpg

Execute:

rename s:/:-:g */*/*.jpg

This results in:

.
├── sub1
│   ├── subsub1
│   └── subsub2
├── sub1-subsub1-001.jpg
├── sub1-subsub1-002.jpg
├── sub1-subsub1-003.jpg
├── sub1-subsub2-001.jpg
├── sub1-subsub2-002.jpg
├── sub1-subsub2-003.jpg
├── sub2
│   ├── subsub1
│   └── subsub2
├── sub2-subsub1-001.jpg
├── sub2-subsub1-002.jpg
├── sub2-subsub1-003.jpg
├── sub2-subsub2-001.jpg
├── sub2-subsub2-002.jpg
└── sub2-subsub2-003.jpg

And you can get rid of the subdirectories using

rm -r sub*/

(final slash is important! it restricts the match to directories)

So finally:

.
├── sub1-subsub1-001.jpg
├── sub1-subsub1-002.jpg
├── sub1-subsub1-003.jpg
├── sub1-subsub2-001.jpg
├── sub1-subsub2-002.jpg
├── sub1-subsub2-003.jpg
├── sub2-subsub1-001.jpg
├── sub2-subsub1-002.jpg
├── sub2-subsub1-003.jpg
├── sub2-subsub2-001.jpg
├── sub2-subsub2-002.jpg
└── sub2-subsub2-003.jpg

If you have a huge collection of files, you cannot process in one call because the file expansion of **/*.jpg will generate a list which is too long (/bin/sh: 1: rename: Argument list too long`). So you have to split the work, for instance by calling rename once by directory:

shopt -s globstar 
for d in **/; do rename s:/:-:g "$d"*.jpg; done

(the / in '**/' is important, it restricts the expansion to directories)

If you still get the Argument list too long message then you have a directory with very many files :) You just have to find a way to split the work, for instance:

for f in path/to/hugedir/A*;jpg; do rename s:/:-:g "$d"*.jpg; done
for f in path/to/hugedir/B*;jpg; do rename s:/:-:g "$d"*.jpg; done
[... etc ...]

If this is still too big and or you want to be blunt, you can also call rename once for each file:

shopt -s globstar 
for f in **/*.jpg; do rename s:/:-:g "$f"; done

(this will work because here the expansion of "**/*.jpg" is just a variable in your shell, and not the list of arguments to a command).

PS: bash assumed here, if you are putting this in a script, make sure the "shebang" is /bin/bash, on some systems sh is a lightweight shell (dash) that doesn't support globstar.

(*) name varies depending on distro, rename for Debian/Ubuntu and derivatives, and IIRC prename on RedHat and derivatives.

Best Answer

Related Solutions

Linux – Recursive, Non-Overwriting File Copy

Linux – rename all jpeg files sequentially in all nested subfolders and move to a new folder

Related Question