SSH – How to Transfer a Named Pipe Using SCP, SFTP, or Rsync

btrfspipescpssh

I intend to transfer full and incremental backups of my btrfs subvolumes to a tape archival service. The service exposes FTP and SSH endpoints. If I were allowed to execute arbitrary commands on the SSH endpoint, then I would do the following to perform an incremental backup:

btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt |
    ssh -p $PORT $USER@$ENDPOINT "cat > $SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg"

I am, however, not allowed to do that:

$ ssh -p $PORT $USER@$ENDPOINT
Last login: Mon Jan  3 01:23:45 2067 from 123.456.789.123

This account is restricted by rssh.
Allowed commands: scp sftp rsync 

If you believe this is in error, please contact your system administrator.

Connection to some.remote.endpoint closed.

So what I thought to do instead was to use the SCP protocol for the transfer. However, my scp binary refuses to transfer a named pipe:

$ scp -P $PORT <(btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt) \
    $USER@$ENDPOINT:$SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg
/dev/fd/63: not a regular file

The irony is that apparently, scp used to do the right thing once. I suppose not everybody thought transferring named pipes could be sane / useful. EDIT (2017-06-03): This was an incorrect observation. As noted by Kenster, the SCP protocol does not permit sending files of unknown size.

UPDATE (2017-06-04): I also tried transferring the data using the SFTP protocol. Apparently, the FTP protocol permits sending files of unknown size, as evidenced by the support for piping data in the ftp (link) and ncftpput (link, section Description, last paragraph) binaries. I found no such support in the SFTP clients I have tried (sftp, lftp), which might be an indication that SFTP (unlike FTP) does not support sending files of unknown size (contrary to what I thought, SFTP is not just FTP tunelled over SSH; it is a different protocol).

UPDATE (2017-06-05): According to the SFTP protocol version 3 internet draft (link), each application-level packet must specify the payload length before the payload (link, Section 3). However, SFTP supports seeks in the written file (link, Section 6.4) with explicit support for writes beyond the current end of file. Therefore, if should be possible to use a small buffer on the client side and send a file of unknown size in small known-sized chunks:

#!/bin/bash
# <Exchange SSH_FXP_INIT requests.>
# <Send an SSH_FXP_OPEN request.>
CHUNK_SIZE=32768
OFFSET=0
IFS=''; while read -r -N $CHUNK_SIZE CHUNK; do
  ACTUAL_SIZE=`cat <<<"$CHUNK" | head -c -1 | wc -c`
  # <Send an SSH_FXP_WRITE request with payload $CHUNK of size
  # $ACTUAL_SIZE at offset $OFFSET.>
  OFFSET=$(($OFFSET+$ACTUAL_SIZE))
done < <(command)
# <Send an SSH_FXP_CLOSE request.>

However, doing the communication manually over the shell would be quite painful. I am looking for an SFTP client that exposes this kind of functionality.

References

Best Answer

lftp 4.6.1 and newer should be able to do this: https://github.com/lavv17/lftp/issues/104

Unfortunately, the command suggested in the linked issue does not work, but a slightly-modified one does:

lftp -p $port sftp://$user:$pass@$host -e "put /dev/stdin -o $filename"

Due to a bug, you must provide something in the password field if you use a SSH agent. But any random string will do.

This command reads from /dev/stdin. If you must use a different named pipe, I imagine that should work too - if not, you can always cat named-pipe | lftp ....

I've tested lftp with a >100GB upload without any issues.

rclone also supports SFTP, and should be able to upload from a piped input with the rcat command, due to be released in 1.38 (next version).

Related Question