Save a single web page (with background images) with Wget

command linedownloadmirroringwebwget

I want to use Wget to save single web pages (not recursively, not whole sites) for reference. Much like Firefox's "Web Page, complete".

My first problem is: I can't get Wget to save background images specified in the CSS. Even if it did save the background image files I don't think –convert-links would convert the background-image URLs in the CSS file to point to the locally saved background images.
Firefox has the same problem.

My second problem is: If there are images on the page I want to save that are hosted on another server (like ads) these wont be included. –span-hosts doesn't seem to solve that problem with the line below.

I'm using:
wget --no-parent --timestamping --convert-links --page-requisites --no-directories --no-host-directories -erobots=off http://domain.tld/webpage.html

Best Answer

From the Wget man page:

Actually, to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to ‘-p’:

wget -E -H -k -K -p http://www.example.com/

Also in case robots.txt is disallowing you add -e robots=off

Related Solutions

Bash – Downloading Complete Web Pages

Have you tried wget --page-requisites ?

       This option causes Wget to download all the files that are neces‐
       sary to properly display a given HTML page.  This includes such
       things as inlined images, sounds, and referenced stylesheets.

How to use WGET to mirror a site 1 level deep, recovering JS, CSS resources including CSS images

wget -p -k http://ExampleSite.com

The -p will get you all the required elements to view the site correctly (css, images, etc). The -k will change all links (to include those for CSS & images) to allow you to view the page offline as it appeared online.

Update: This is specific for your example site: tumblr.com

wget -H -N -k -p --exclude-domains quantserve.com --no-check-certificate -U "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110613 Firefox/6.0a2" https://www.tumblr.com

The Breakdown:

-H = Allows wget to go to span a foreign host. Required since tumblr does not have its images on the front page on the same address, they are using secure.assets.tumblr.com see note on excluding domains

-N = will grab only files that are newer that what you currently have, in case you are downloading the same page again over time

-k = convert your links to view it offline properly

-p = grabs all required elements to view it correctly (css, images, etc)

--exclude-domains = since the tumblr.com homepage has a link for quantserve.com and i'm guessing you don't want this stuff, you need to exclude it from your wget download. Note: This is a pretty important one that you should use with -H because if you go to a site and they have multiple links for outside hosts (think advertisers & analytics stuff) then you are going to grab that stuff also!

--no-check-certificate required since tumblr is using https

-U changes the user-agent. Not really necessary in this instance since it allows the default wget user-agent but I know some sites will block it. I just threw it in here so in case you run into any problems on other sites. In the example snippet I gave, it appears as Mozilla Firefox 6.02a

finally you have the site: https://www.tumblr.com

Best Answer

Related Solutions

Bash – Downloading Complete Web Pages

How to use WGET to mirror a site 1 level deep, recovering JS, CSS resources including CSS images

Related Question