Wget files by pattern only from specified directories recursively

wget

I need to download on an hourly basis (sometimes more frequently), files which are being written in segments of 24 hours. The files I am interested in are in specific subdirectories which I am trying to specify with -I list but this doesn't work for some reason.

If I don't specify directories the files I need download fine with the -A acclist option but I end up with lots of empty directories that are being created because they exist on the host.

my current line reads:

wget -np -nH --cut-dirs=X -c -N -r -l 0 \
     -I /dir1,/dir2,...,/some_dir -A acclist \
     http://hostname/X_sub_directories/

How do I download only the files I want and create only the directory hierarchy for those files?

Best Answer

you could add a post process command to wipe out the empty directories created.

wget -np -nH --cut-dirs=X -c -N -r -l 0 \
     -I /dir1,/dir2,...,/some_dir -A acclist \
     http://hostname/X_sub_directories/    \
     &&  find -depth -type d -empty -exec rmdir {} \;

Related Solutions

Wget Recursive Download – How to Download Recursively with Wget

wget by default honours the robots.txt standard for crawling pages, just like search engines do, and for archive.org, it disallows the entire /web/ subdirectory. To override, use -e robots=off,

wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/

Wget: downloading files selectively and recursively

This command will download only images and movies from a given website:

wget -nd -r -P /save/location -A jpeg,jpg,bmp,gif,png,mov "http://www.somedomain.com"

According to wget man:

-nd prevents the creation of a directory hierarchy (i.e. no directories).

-r enables recursive retrieval. See Recursive Download for more information.

-P sets the directory prefix where all files and directories are saved to.

-A sets a whitelist for retrieving only certain file types. Strings and patterns are accepted, and both can be used in a comma separated list (as seen above). See Types of Files for more information.

If you would like to download subfolders you need to use the flag --no-parent, something similar to this command:

wget -r -l1 --no-parent -P /save/location -A jpeg,jpg,bmp,gif,png,mov "http://www.somedomain.com"

-r: recursive retrieving
-l1: sets the maximum recursion depth to be 1
--no-parent: does not ascend to the parent; only downloads from the specified subdirectory and downwards hierarchy

Regarding the index.html webpage. It will be excluded once the flag -A is included in the command wget, because this flag will force wget to download specific type of files, meaning if html is not included in the list of accepted files to be downloaded (i.e. flag A), then it will not be downloaded and wget will output in terminal the following message:

Removing /save/location/default.htm since it should be rejected.

wget can download specific type of files e.g. (jpg, jpeg, png, mov, avi, mpeg, .... etc) when those files are exist in the URL link provided to wget for example:

Let's say we would like to download .zip and .chd files from this website

In this link there are folders and .zip files (scroll to the end). Now, let's say we would like to run this command:

wget -r --no-parent -P /save/location -A chd,zip "https://archive.org/download/MAME0.139_MAME2010_Reference_Set_ROMs_CHDs_Samples/roms/"

This command will download .zip files and at the same time it will create an empty folders for the .chd files.

In order to download the .chd files, we would need to extract the names of the empty folders, then convert those folder names to its actual URLs. Then, put all the URLs of interest in a text file file.txt, finally feed this text file to wget, as follows:

wget -r --no-parent -P /save/location -A chd,zip -i file.txt

The previous command will find all the chd files.

Best Answer

Related Solutions

Wget Recursive Download – How to Download Recursively with Wget

Wget: downloading files selectively and recursively

Related Question