Recursive scp without following links or creating a giant tar file

scpsymlink

So I did a recursive scp on my remote fileserver (in another state) and it created an infinite loop of links on my remote web directory…

http://www.linuxquestions.org/questions/linux-general-1/recursive-scp-w-o-following-links-658857/ says that I can can try creating a giant tar file. There is a problem with this though – I'm running the recursive scp on a Linux machine in my office, and I'm copying the files all to my external hard drive, which is in FAT32 format (because I need something that's readable by both UNIX and Windows). FAT32 doesn't support large filesizes. So I would have to try something different.

There's also a rsync option but the Linux machine in my office is very primitive (it's igel) so it doesn't have rsync…

Best Answer

I would not recommend using scp for transferring large file trees directly, because it does not handle neither hard nor soft links properly, also the stream is not compressed.

I'd recommend cpio with (de)compression on the fly:

ssh user@host "cd /path/to/files && find . | cpio -ov | bzip2 -c" | bunzip2 -c | cpio -ivd

Also, find can handle additional conditions, like "files must be less than 4G"

find . -size -4G | ...

To make cpio more space-friendly (to handle spaces in file names properly) use

find . -print0 | cpio -0 -ivd | ...

Related Solutions

SCP – SCP Fails Without Error

Ok LOL, I just figured out what the problem is.

Since I like cows so much, I've put fortune | cowsay at the top of my .bashrc file which produces output like the following when starting bash:

 _______________________________________
< You will lose an important disk file. >
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

This is all fine (and sometimes funny) when running bash interactively. However, bash reads ~/.bashrc when it is interactive and not a login shell, or when it is a login shell and its parent process is rshd or sshd. When you run scp, the server starts a shell which starts a remote scp instance. The output from .bashrc confuses scp because it is sent the same way the scp protocol data is sent. This is apparently a known bug, see here for more details.

Also note that the underscores I mentioned in the question are those in the top line of the text balloon.

So the solution was simple: I put the following at the top of .bashrc on the remote (destination) machine:

# If not running interactively, don't do anything
[[ $- == *i* ]] || return

This line is present in the default .bashrc but was put way down because of my many (apparently careless) edits.

Copying large tree from one machine to another, maintaining ownership

As root, set up a named pipe:

# mkfifo /tmp/fifo
# chmod o+w /tmp/fifo

Then, transfer your data as me:

$ tar cfzp - foldertocopy | ssh me@machine "cat > /tmp/fifo"

But read it as root:

# tar -xfzp /tmp/fifo

Best Answer

Related Solutions

SCP – SCP Fails Without Error

Copying large tree from one machine to another, maintaining ownership

Related Question