What program should I use to transfer 20TB data across the network

data transferrsyncthunderbolt

I need to copy 20TB of data onto a thunderbolt array. The box where the data exists does not have a thunderbolt connection, so I will need to utilize the local 1GB network for this. (Yes, it will take forever).

I tried to use Filezilla/sftp, but it crashed when the queue grew super large. Is rsync or scp the way to go?

Best Answer

rsync is a good way to go (scp is pretty much the same with fewer features). You may want to use the -Z option, which will enable zlib compression. Depending on how fast your drives/computer are, it may be faster than sending uncompressed, i.e. if your network link is saturated. You may also want the archive mode option, -a which will preserve symlinks, permissions, and creation/modification times, as well as copy directories recursively. Depending on what you're copying you might want -E which preserves extended attributes and mac resource forks. Finally, --progress will show you progress information.