How to use wget to download all links from the site and save to a text file

As jayhendren suggested, I had tried listing the domain bp.blogspot.com on the list following the -D flag. However what I forgot to do is add the -H flag. Why wget requires the extra -H flag to be added separately from the list of domains to follow with the -D flag is unclear to me, but it works. Here is the command I ultimately specified to mirror the Blogger site including the images served from the external domain:

wget --domains=blogspot.it,bp.blogspot.com -H --mirror -e robots=off \
  --wait 0.5 --convert-links http://yoursitehere.blogspot.it/

Note: this works from Italy. Convert .it to .com or to whatever other top-level domain if you want this to work from your location.

Regards.

Rather than try and do this using wget I'd suggest using a more appropriate tool for downloading complex "sets" of files or filters.

You can use httrack to download either entire directories of files (essentially mirror everything from a site) or you can specify to httrack a filter along with specific file extensions, such as download only .pdf files.

You can read more about httrack's filter capability which is what you'd need to use if you were interested in only downloading files that were named in a specific way.

Here are some examples of the wildcard capability:

• *[file] or *[name] - any filename or name, e.g. not /,? and ; characters
• *[path] - any path (and filename), e.g. not ? and ; characters
• *[a,z,e,r,t,y] - any letters among a,z,e,r,t,y
• *[a-z] - any letters
• *[0-9,a,z,e,r,t,y] - any characters among 0..9 and a,z,e,r,t,y

Example

$ httrack http://url.com/files/ -* +1_[a-z].doc -O /dir/to/output

The switches are as follows:

• -* - remove everything from list of things to download
• +1_[a-z].doc - download files named 1_a.doc, 1_b.doc, etc.
• -O /dir/to/output - write results here