How to download all images from a website using wget

downloadwget

I want you to help me with wget, I want to download all images from stock images website like https://pixabay.com/ but when I enter the code in the terminal nothing is downloading, no jpg, no zip.

I used this code:

wget -r -A jpg https://pixabay.com/

I sometimes use jpg or zip depending on the website. I have tried with more websites:

http://www.freepik.com/
http://www.freeimages.com/
http://all-free-download.com/

It's not downloading at all.

Best Answer

Here is the working command:

wget -U "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0" -nd -r --level=1  -e robots=off -A jpg,jpeg -H http://pixabay.com/
  • -U "..." : The website is returning HTTP error 403 (forbidden) as it only allows a given list of User-Agent to access their pages. You have to stipulate an User-Agent of a common browser (firefox, chrome, ...). The one I gave you is a working example.
  • -nd (no-directories) from man: "Do not create a hierarchy of directory when retrieving recursively."
  • -e robots=off: do not follow robot.txt exclusion
  • -H: enable retrieving files across hosts (here pixabay.com and cdn.pixabay.com are considered as different hosts)

if there is some rate limit mechanism, add the following option --wait 1

Related Question