File Synchronization – Best Way to Sync Files to a VFAT Partition

file-copysynchronizationvfat

POSIX filenames may contain all characters except /, but some filesystems reserve characters like ?<>\\:*|". Using pax, I can copy files while replacing these reserved characters:

$ pax -rw -s '/[?<>\\:*|\"]/_/gp' /source /target

But pax lacks an --delete option like rsync and rsync cannot substitute characters. I'm looking for a simple way to backup my music collection to an external hard drive on a regular basis.

Best Answer

You can make a view of the FAT filesystem with POSIX semantics, including supporting file names with any character other than / or a null byte. POSIXovl is a relatively recent FUSE filesystem for this.

mkdir backup-fat
mount.posixovl -S /media/sdb1 backup-fat
rsync -au /source backup-fat/target

Characters in file names that VFAT doesn't accept are encoded as %(XX) where XX are hexadecimal digits. As of POSIXovl 1.2.20120215, beware that a file name like %(3A) is encoded as itself, and will be decoded as :, so there is a risk of collision if you have file names containing substrings of the form %(XX).

Beware that POSIXovl does not cope with file names that are too long. If the encoded name doesn't fit in 255 characters, the file can't be stored.

POSIXovl stores unix permissions and ownership in files called .pxovl.FILENAME.

Related Solutions

Linux – Why doesn’t the Linux vfat driver allow certain characters

If I'm not mistaken any filesystem supported by Windows (FAT/VFAT/FAT32/NTFS) should meet some standard naming conventions for valid file and directory names. The aforementioned list of characters are reserved characters which can't be used even with Unicode characters.

I would say that Linux implementation of fat/vfat filesystems tries to be compliant with this kind of "Windows standard". Perhaps, this article can bring more details regarding it.

Best way to sync files – copy only EXISTING files and only if NEWER than target

I believe you can use rsync to do this. The key observation would be in needing to use the --existing and --update switches.

        --existing              skip creating new files on receiver
        -u, --update            skip files that are newer on the receiver

A command like this would do it:

$ rsync -avz --update --existing src/ dst

Example

Say we have the following sample data.

$ mkdir -p src/; touch src/file{1..3}
$ mkdir -p dst/; touch dst/file{2..3}
$ touch -d 20120101 src/file2

Which looks as follows:

$ ls -l src/ dst/
dst/:
total 0
-rw-rw-r--. 1 saml saml 0 Feb 27 01:00 file2
-rw-rw-r--. 1 saml saml 0 Feb 27 01:00 file3

src/:
total 0
-rw-rw-r--. 1 saml saml 0 Feb 27 01:00 file1
-rw-rw-r--. 1 saml saml 0 Jan  1  2012 file2
-rw-rw-r--. 1 saml saml 0 Feb 27 01:00 file3

Now if I were to sync these directories nothing would happen:

$ rsync -avz --update --existing src/ dst
sending incremental file list

sent 12 bytes  received 31 bytes  406.00 bytes/sec
total size is 0  speedup is 0.00

If we touch a source file so that it's newer:

$ touch src/file3 
$ ls -l src/file3
-rw-rw-r--. 1 saml saml 0 Feb 27 01:04 src/file3

Another run of the rsync command:

$ rsync -avz --update --existing src/ dst
sending incremental file list
file3

sent 115 bytes  received 31 bytes  292.00 bytes/sec
total size is 0  speedup is 0.00

We can see that file3, since it's newer, and that it exists in dst/, it gets sent.

Testing

To make sure things work before you cut the command loose, I'd suggest using another of rsync's switches, --dry-run. Let's add another -v too so rsync's output is more verbose.

$ rsync -avvz --dry-run --update --existing src/ dst 
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
file1
file2 is uptodate
file3 is newer
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 88 bytes  received 21 bytes  218.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)