MacOS – How to download images from an xml feed using terminal

applescriptbashcommand linemacosterminal

How can I download png images from an xml feed that looks the following way?

<ListBucketResult xmlns="http://url1.com">
  <Name>imagesToUse</Name>
<Prefix/>
<Marker/>
<IsTruncated>false</IsTruncated>
<Contents>
  <ImageStuff>imagetodownload.png</ImageStuff>
  <Size>17293</Size>
</Contents>

I was taking a look at this question on here about the same topic, except that I don't need to rename them, just download them.

Being new to shell scripts, I'm not quite sure how to do this (although I can definitely do this in 2 minutes in objective-c).

I have tried creating the following script and running bash myscript.sh

#!/bin/bash
URL='https://theurlwiththexml.com'
i=0
for u in $(curl -s "$URL" |
           grep png | 
           sed -E 's/.*<ImageStuff>(.+)<\/ImageStuff>/\1/'); do
    curl -s "$u" -o $i.png
    (( i++ ))
done

But nothing happens. It just prints another line without errors.

Best Answer

After curl http://theurlwiththexml.com:

sed -n 's,.*<ImageStuff>\(.*\.png\)<.*,\1,p'|xargs -I, curl http://url1.com/, -o ,

sed -n disables printing, \( and \) are a portable way to specify groups, and p prints the pattern space. xargs -I specifies a replacement string and curl -o specifies an output file.

Or using xmlstarlet, parallel, and wget:

xml sel -t -v //ImageStuff|parallel wget http://url1.com/{}

sel is an abbreviation for select, -t -v uses the value template, and //ImageStuff is an XPath expression that matches all ImageStuff nodes.

Related Solutions

Curl – How to Use Curl to Download Images from Website Using Wildcard

You can use this bash code for the actual URL you provided in your comment.

for i in $(for j in {1..8}; do
    curl http://occipital.com/user/01ca-255/george-coghill/$j/;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  curl -L -o "${i##*/}" "$i";
done

Feel free to write this command as one line; all line breaks were only added to increase legibility. You may copy or remove them, whatever you like.

What this code does:

Iterate over the 8 pages of your account gallery
Extract the image names from the preview images
Fetch all full-sized images using this list of names

If you want to only download files which don't exist, and don't know the number of gallery pages up front, you can adjust the code to this:

for i in $(
  j=1;
  while curl --max-redirs 0 -L -f \
      http://occipital.com/user/01ca-255/george-coghill/$((j++))/; do
    :;
  done \
  | sed -n 's,.*/viewer/\(......_flat\)_small\.jpg.*,http://occipital.com/images/viewer/\1.jpg,p'
); do
  [[ -f "${i##*/}" ]] || curl -L -o "${i##*/}" "$i";
done

The first code now increments the page number until a redirect or error occurs. The server will redirect you to the last existing page if you try to visit a page after the existing ones. The [[ -f … ]] || part will execute curl only if the corresponding file does not exist yet.

Using cURL to download images in XML feed

If a shell script solution is acceptable as well, try the following

#!/bin/bash
URL='http://tweetriver.com/hud_ro/steer_filtered.xml?include_entities=1'
i=0
for u in $(curl -s "$URL" |
           grep jpg | 
           sed -E 's/.*<profile_image_url>(.+)<\/profile_image_url>/\1/'); do
    curl -s "$u" -o $i.jpg
    (( i++ ))
done

This of course only works for this specific XML example and requires all files to be in jpg format.

Best Answer

Related Solutions

Curl – How to Use Curl to Download Images from Website Using Wildcard

Using cURL to download images in XML feed

Related Question