Download tool needed — with custom headers, resume, retry, custom filename output, and dynamic redirection

downloadwebwget

Simple problem — I need a download manager, console, GUI, does not matter. The catch is in the features:

  • custom header — I have to pass a cookie in order to download (authorization)

  • resume — so if download stops for any reason, I could continue from 50% (for example), not from start

  • retry — so the program could automatically try establish connection and downloading again

  • custom filename output — so I could give specific filename, where the data go

  • and dynamic redirection — so if I ask server for file X, and server answers "OK, here is X?token=767" downloader will accept this with original name — X (i.e. it will think it is the same I asked for)

Since I prefer console, so far I tried wget, curl, aria2c. All of them fail. wget does not handle custom filename output (it is just a dump to a file) and it is "fooled" by redirection — i.e. as with above example it will store the data in file X?token=767 which in effect ruins both resume and retry, because with each reconnection the token changes.

aria2c and curl do not even start downloading. My guess is, they don't accept redirection at all, aria2c retries again and again with 0 bytes download and no success, curl downloads the response from server the file was moved to X?token=767.

For the record, below my commands for downloading:

wget

wget -rc --no-cookies --header "Cookie: COOKIEKEY=COOKIEVALUE" \
     "https://www.foo.bar/file.gz"

aria2c

aria2c -o f.gz --check-certificate=false -c -m0 \
       --header="Cookie: COOKIEKEY=COOKIEVALUE" \
       "https://www.foo.bar/file.gz"

curl

curl -o f.gz -C - --retry 1000 \
     -H "Cookie: COOKIEKEY=COOKIEVALUE" \
     "https://www.foo.bar/file.gz"

Best Answer

The wget redirection problem can be solved by using

wget --trust-server-names http://www.example.com/X?1234
Related Question