In what order does sftp fetch files when using “get -r folder”

file-copyfile-transfersftp

I'm doing a file transfer using sftp. Using the get -r folder command, I'm surprised about the order that the program is downloading the content.

It looks like it would be selecting the files it needs to download randomly. I can't believe that this is actually the case and I'm asking myself what's going on behind the scenes?

What's the order that sftp follows when downloading a folder with its content?

From what I can see so far, it is not by name nor by size.

Best Answer

When you list the directory contents with the ls command, it will sort the listing into alphanumeric order according to current locale's sorting rules by default. It is easy to assume that this is the "natural order" of things within the filesystem - but this isn't true.

Most filesystems don't sort their directories in any way: when adding a new file to a directory, the new file basically gets the first free slot in the directory's metadata structure. Sorting is only done when displaying the directory listing to the user. If a single directory has hundreds of thousands or millions of files in it, this sorting can actually require non-trivial amounts of memory and processing power.

When the order in which the files are processed does not matter, the most efficient way is to just read the directory metadata in order and process the files in the order encountered without any explicit sorting. In most cases this would mean the files will be processed basically in the order they were added to the directory, interspersed with newer files in cases where an old file was deleted and a later-added file reclaimed its metadata slot.

Some filesystems might use tree structures or something else in their internal design that might enforce a particular order for their directory entries as a side effect. But such an ordering might be based on inode numbers of the files or some other filesystem-internal detail, and so would not be guaranteed to be useful for humans for any practical purpose.

As @A.B said in the question comments, a find command or a ls -f or ls --sort=none would list the files without any explicit sorting, in whatever order the filesystem stores its directories.

Related Solutions

Ssh – How to copy a file that is still being written over ssh

For a single file instead of using SFTP you could pipe the file over ssh using cat or pv at the sending side and using tee on the middle server to both send the data to a file there and send a copy over the another ssh link the other side of which just writes the data to a file. The exact voodoo required I'll leave as an exercise for the reader, as I've not got time to play right now (sorry). This method would only work if the second destination is publicly accessible via SSH which may not be the case as you describe it as a client machine.

Another approach, which is less "run and wait" but may otherwise be easier, it to use rsync between the server and client B. The first time you run this it may get a partial copy of the data, but you can just re-run it to get more data afterwards (with one final run once the Client1->Server transfer is complete). This will only work if the server puts the data direct into the right file-name during the SFTP transfer (sometimes you will see the data going into a temporary file which is then renamed once the file is completely transferred - this is done to make the file update more atomic but will render the rsync idea unusable). You could also use rsync for the C1->S transfer instead of scp (if you use the --inplace option to avoid the problem mentioned above) - using rsync would also give you protection against needing to resend everything if the C1->Server connection experiences problems during a large transfer (I tend to use rsync --inplace -a --progress <source> <dest> instead of scp/sftp when rsync is available, for this "transfer resume" behaviour).

To summarise the above, running:

rsync --inplace -a --progress <source> user@server:/<destination_file_or_folder>

on client1 then running

rsync --inplace -a --progress user@server:/<destination_file_or_folder> <destination_on_cli2>

on client2 repeatedly until the first transfer is complete (then running once more to make sure you've got everything). rsync is very good at only transferring the absolute minimum it needs to update a location instead of transferring the whole lot each time. For paranoia you might want to add the --checksum option to the rsync commands (which will take much more CPU time for large files but won't result in significantly more data being transfered unless it is needed) and for speed the --compress option will help if the data you are transferring is not already in a compressed format.

Renaming files in SFTP session

Works for me, so you will need to provide more information about the problem:

chris@localhost$ finger 2> file.txt
chris@localhost$ sftp remotehost
Connected to remotehost.
sftp> ls -l file.txt
Can't ls: "/home/chris/file.txt" not found
sftp> ls -l file.tmp
Can't ls: "/home/chris/file.tmp" not found

# So the file doesn't exist on the remote in either form

sftp> put file.txt file.tmp
Uploading file.txt to /home/chris/file.tmp
file.txt                                                      100%  501     0.5KB/s   00:01
sftp> ls -l file.txt
Can't ls: "/home/chris/file.txt" not found
sftp> ls -l file.tmp
-rw-r-xr--    0 1001     1001          501 Aug 12 16:35 file.tmp

# It has arrived as file.tmp

sftp> rename file.tmp file.txt
sftp> ls -l file.txt
-rw-r-xr--    0 1001     1001          501 Aug 12 16:35 file.txt
sftp> ls -l file.tmp
Can't ls: "/home/chris/file.tmp" not found

# And been successfully renamed

Best Answer

Related Solutions

Ssh – How to copy a file that is still being written over ssh

Renaming files in SFTP session

Related Question