Curl – How to Use Curl to Download Images from Website Using Wildcard

command lineterminal

Is there a way to use curl to download non-sequential images with a wildcard? I want to download all the panoramas I upload to my account at 360 Panorama to a local folder so I don't have to do this manually each time.

The images files follow the pattern of: http://occipital.com/images/viewer/XYZ_flat.jpg with the XYZ being random. It also seems from that URL that I need to be logged in or seen as logged in to the site.

Best Answer

You can use this bash code for the actual URL you provided in your comment.

for i in $(for j in {1..8}; do
    curl http://occipital.com/user/01ca-255/george-coghill/$j/;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  curl -L -o "${i##*/}" "$i";
done

Feel free to write this command as one line; all line breaks were only added to increase legibility. You may copy or remove them, whatever you like.

What this code does:

  1. Iterate over the 8 pages of your account gallery
  2. Extract the image names from the preview images
  3. Fetch all full-sized images using this list of names

If you want to only download files which don't exist, and don't know the number of gallery pages up front, you can adjust the code to this:

for i in $(
  j=1;
  while curl --max-redirs 0 -L -f \
      http://occipital.com/user/01ca-255/george-coghill/$((j++))/; do
    :;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  [[ -f "${i##*/}" ]] || curl -L -o "${i##*/}" "$i";
done

The first code now increments the page number until a redirect or error occurs. The server will redirect you to the last existing page if you try to visit a page after the existing ones. The [[ -f … ]] || part will execute curl only if the corresponding file does not exist yet.

Related Question