Why does this curl command from firefox not download anything

curlfirefox

I opened https://raw.githubusercontent.com/andreafrancia/trash-cli/master/README.rst in Firefox, and copy the following curl command from Tools->Web Developer->Network:

curl 'https://raw.githubusercontent.com/andreafrancia/trash-cli/master/README.rst' -H 'Host: raw.githubusercontent.com' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-GB,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'If-None-Match: "6931c3b4d0e94743bb93a36ed8e8c3f5add12f9a"' -H 'Cache-Control: max-age=0'

When I run it in lxterminal, it doesn't download anything, even if I add -O to it. I was wondering why it doesn't download, and how I can make it download the file?

Thanks.

Best Answer

When debugging curl issues, the -v option is often helpful. In this particular instance, you’re running afoul of the If-None-Match header, which tells the server that you already have the file matching “6931c3b4d0e94743bb93a36ed8e8c3f5add12f9a” and that you’re not interested in retrieving it again if it hasn’t changed. -v shows you this by indicating that the server responds with a 304 header:

< HTTP/1.1 304 Not Modified

To download your file, drop the header:

curl 'https://raw.githubusercontent.com/andreafrancia/trash-cli/master/README.rst' -H 'Host: raw.githubusercontent.com' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-GB,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Cache-Control: max-age=0'

In this particular instance you’ll get the same result with

curl 'https://raw.githubusercontent.com/andreafrancia/trash-cli/master/README.rst'

Related Solutions

How to download videos from Coursera with curl

As with any web service, the exact method changes a lot.

In the coursera-dl project, we try to do all the magic. The code involves jumping through some redirects and keeping cookies in place (which changes almost at every login), but you can run it with the --debug option to see how it calls curl or wget or your preferred downloader.

Disclaimer: I am a contributor to the project.

Extract download progress from curl output

The main issue is that sed works on lines so it doesn't do anything until the first \n is reached and that doesn't happen until your command is finished. You can get around this by exchanging \rs with \ns:

$ curl --progress-bar http://127.0.0.1/test.tar.bz2 -o test.tar.bz2 2>&1 | 
   tr $'\r' $'\n' | sed -r 's/[# ]+/#/g;'

This, however, brings you up against buffering, sed will now act on groups of lines. The final solution I hacked together was to redirect the error to a file and then deal with that file:

$ curl --progress-bar http://127.0.0.1/test.tar.bz2 -o test.tar.bz2 2>er
$ while :; do 
    echo -ne "$(tr $'\r' $'\n' < er | tail -n 1 | sed -r 's/^[# ]+/#/;')\r"; 
  done

The command above will parse the error file (er) and print the result with an \r making it update constantly. You will need to break out of it manually.

Suggestion from an anonymous user: You can also put stdbuf -oL in front of tr and sed thus modifying the buffering behaviour of those commands.

Best Answer

Related Solutions

How to download videos from Coursera with curl

Extract download progress from curl output

Related Question