Upload only few bytes from a file to the SFTP

linuxsftpsshupload

I have a use case wherein I need to upload only some bytes(can be in GB) of the file to SFTP server rather than the entire file. With SFTP PUT command the entire file gets uploaded, instead of some bytes.

I do not want to read the file and then make a smaller file out of that and then upload that entire smaller file to SFTP. This can be very memory intensive if the bytes to be uploaded are in GB. So, discarding this option.

Is there any other way to do this using SFTP(may be using -o to use ssh options) command. Perhaps something like scp: How to copy only the last 10% of a file?

pls suggest

Best Answer

The standard OpenSSH sftp client doesn't support partial uploads.

If the server is running regular Linux, then you can directly pipe data to an SSH connection without using SFTP:

$ tail -c 2G ~/bigfile | ssh myserver "cat > smallchunk"

(Here tail -c 2G is just an example command that outputs the last 2 GB of the file. Replace it as appropriate – if you want the first X bytes then you'll want to use head, if you want want bytes from the middle use some combination of head|tail or maybe dd.)

Optional improvement: You can insert 'pv' to display a progress bar.

$ tail -c 2G ~/bigfile | pv | ssh myserver "cat > smallchunk"

If you do require SFTP, then the lftp client can be tricked into achieving this by uploading a named pipe:

Create a named pipe somewhere:
```
$ mkfifo /tmp/pipe
```
Run a command that gets whichever bytes you want, and writes them to the named pipe:
```
$ tail -c 2G ~/bigfile > /tmp/pipe &
```
The command will immediately be paused as soon as it tries to write to the pipe. (This works exactly like the | nameless pipe in the previous example – the writing process gets paused until the reading process reads everything out of the pipe, 4kB at a time. Hence the & to let it happen in background.)

Ask lftp to upload the pipe:

$ lftp sftp://myserver
lftp> put -o smallchunk /tmp/pipe
2G bytes transferred

For clarity regarding pipes: Here's example 1 again...

$ tail -c 2G ~/bigfile | ssh myserver "cat > smallchunk"

...and here's example 1 rewritten to use 'mkfifo' from example 2:

$ mkfifo /tmp/pipe
$ tail -c 2G ~/bigfile > /tmp/pipe &
$ ssh myserver "cat > smallchunk" < /tmp/pipe

Both of these work the same way: they use a pipe connecting one writer process to one reader. Both named and nameless pipes have a 4 kB buffer; every time it fills up, the writer is paused until the reader gets the data.

Related Solutions

Using SCP or SFTP with the ssh config file

From the help text: "... [-F ssh_config] ..."

According to the above, -F expects one argument: path to an OpenSSH configuration file, ~/.ssh/config or similar. But you are giving it a gzipped SQL dump instead.

Since plain ssh myalias is already working, you don't even need the -F option here. Just sftp myalias would connect to the server.
However, the OpenSSH sftp client does not support uploading files like you are trying to; it can only download files (using the syntax host:path) or work in interactive mode. For uploading, you need to use either the interactive mode...
```
$ sftp myalias
sftp> cd /tmp
sftp> put db.sql.gz
```
...or the scp tool:
```
scp db.sql.gz myalias:/tmp
```
or
```
scp db.sql.gz webost@staging2.example.com:/tmp
```

(sftp does have a batch mode in which it can read commands from a file, using -b, but it is simpler to use scp for single uploads.)

There are other SFTP clients as well – lftp is good for interactive use, while curl can be easier to automate. For backups and such, you could also use rsync (which runs its own protocol but still inside SSH).

Linux: prevent SFTP users from deleting FTP’d files even straight after they upload them

Apologies i do not have sufficent reputation here to post all the links inline. I have prepared a gist which preserves them: https://gist.github.com/3590779

There's no way to achieve this with simple permissions. Due to OpenSSH's sftp-server you won't be able to implement the full requirements list but depending on the filesystem the files are being uploaded to, you can leverage [attributes] and [ACLs] to achieve some of your requirements.
Yes, the [sftp-server takes a -u parameter] (you can set this in your [sshd_config] on the Subsystem sftp line) which sets the umask for all uploads.
Yes, you can make use of inotify, one way may be with the [incron] tool although there are many ways to use inotify. Inotify allows you to have the kernel notify a userspace program on a filesystem event you identify, e.g. adding a file to a directory. You can then run a command on this event.

(3 Part 2) An alternative approach that may not be suitable for you is to use something like vsftpd with SSL protected FTP. This allows for encrypted FTP but because vsftpd is a full featured FTP server it provides simple configuration (see the [chown_uploads parameter] in vsftpd.conf) to match your exact needs.
Yes, via the inotify subsystem, you could register a watch for the [IN_CLOSE_WRITE] event.

Best Answer

Related Solutions

Using SCP or SFTP with the ssh config file

Linux: prevent SFTP users from deleting FTP’d files even straight after they upload them

Related Question