Linux – Is it possible to download extremely large files intelligently or in parts via SSH from Linux to Windows

So, does the "pause" feature really work?

Yes, normally it does.

It works as follows: Whenever you download a file, you send a HTTP request to the server with the file in question. The server responds with a HTTP message, which consists of a header and the actual content.

If the size of the file requested is known, the HTTP header reveals the "Content Length" to your browser.

For example, I'm trying to download a PDF file, and here's the response:

charon:~ werner$ curl -I www.ready.gov/business/_downloads/sampleplan.pdf
HTTP/1.1 200 OK
Server: Apache
ETag: "230b73353fc7715f06267967df11be04:1241094925"
Last-Modified: Wed, 29 Apr 2009 20:56:46 GMT
Accept-Ranges: bytes
Content-Length: 293125
Content-Type: application/pdf
Date: Wed, 07 Sep 2011 14:49:33 GMT
Connection: keep-alive

What's important about this are the Content-Length and the Accept-Ranges fields.

• Accept-Ranges means that you can access the file part-by-part, if needed
• Content-Length tells you the whole size of the file

Now, when you start a download, your browser will download the file as usual, but it will of course also keep track of the bytes downloaded and store everything in a temporary file. If you then click "pause", the connection will just be aborted.

However, since the browser knows the number of downloaded bytes, when you click "resume", it can request the file download to be continued at exactly this point, with the HTTP Range field. This is all explained in the HTTP 1.1 Header Field Definitions:

HTTP retrieval requests using conditional or unconditional GET methods MAY request one or more sub-ranges of the entity, instead of the entire entity, using the Range request header, which applies to the entity returned as the result of the request.

The tricky thing is that when your connection is prone to errors, Chrome might not realize that there was a connection loss, therefore record a wrong number of downloaded bytes or even fail to resume the connection to the server. I don't know about the Chrome internals of doing this, but it might not be able to resume a download if it can't send a successful HTTP request.

According to this answer, Chrome could theoretically consider a download as "finished" even though the TCP connection was closed/aborted manually. This would explain the "seems as if it's working" you've described.

Also, some servers might not support the Range command, although I think this is rare. Some sites like Rapidshare seem to make it impossible to resume some downloads.

Finally, you might consider using a download manager and see if that resolves your problems. Other than that, using BitTorrent to download files if possible is probably the safer option than a plain HTTP download.

The CLI tools cURL and Wget both have resume which will pick up where they left off, assuming the HTTP server itself supports HTTP range requests.

curl -C - http://...
wget --continue http://...

I have found both to be reliable. And with the bandwidth limiting feature it means you can set long-running downloads to use less than your average bandwidth so the connection is still usable for other purposes.

There's no HTTP solution for corrupt downloads, HTTPS is more likely to notice certain problems though. The only thing you can do is verify checksums or signatures at the end of the download.

The simplest protocol I'm aware of that could solve your problem is BitTorrent, with rsync a close second.