Download the directory structure of a website without its files

wget

I googled for this but couldn't find an example of how to use eg. wget to download the directory structure of a website without downloading any of its files.

At this point, I just need to understand how a website is organized. I'll mirror the site later.

Best Answer

At the command prompt type:

wget -r --spider www.your-website.com

Alternate command with an option to specify the maximum depth level depth.

wget -r --spider -l depth www.your-website.com

Recursive retrieval options:

-r  
--recursive
    Turn on recursive retrieving.    The default maximum depth is 5.

--spider
    Don't download anything

-l depth
--level=depth
    Specify recursion maximum depth level depth.

Related Solutions

How to find out the real download URL on download sites that use redirects

When you've started the transfer in Firefox, cancel it, and right click the download and hit "Copy download link". If you're using an older version, right click it and click on "Properties", and the link will be in the window which opens.

In Chrome - run download as normal - then go to Menu - Downloads - and you should see the direct link which was used. Or press Ctrl + J to open the window.

Windows – Using wget with windows command prompt to download the files outside the site directory

How do I do download the file outside my site directory?

You can't ask the web server to provide it, for security reasons.

Try using ftp instead

wget ftp://username:password@www.example.com/data.csv

If FTP service isn't available, configure it or use sftp instead, for greater security.

Best Answer

Related Solutions

How to find out the real download URL on download sites that use redirects

Windows – Using wget with windows command prompt to download the files outside the site directory

Related Question