Download multiple URLs at once

downloadwget

I'm aware of wget -i as a way to download a list of URLs. The only trouble is that I need to pass some different POST data to each one, which works for single urls using wget --post-data= but not for lists.

I'm open to any CLI downloader, or even something in JS or Python. I would however like to get either a progress bar for each download or a log file updated each time a dl finishes, or some other way of knowing when a dl finishes.

Best Answer

If you already have a list of URLs, just add the POST data to that list. Something like:

www.example.com    postdata1
www.foo.com
www.bar.com       second_post_data

Then, instead of using -i, read the file in a bash loop and pass the data to wget:

while read url post; do wget --post-data="$post" $url; done < list.txt

To run them in parallel so that multiple files are downloaded at the same time, use & instead of ;. Careful though, this will launch a separate wget process for each URL.

while read url post; do wget --post-data="$post" $url & done < list.txt

One trick I use for launching this type of thing is keeping track of how many are currently running and only running the next one if the number is below a threshold, 10 for example:

while read url post; do 
 while [[ "$(pgrep -fc wget)" -gt 9 && $c -gt 10 ]]; do sleep 1; done; 
 ((c++));  
 echo "Launching $url ($c)"; 
 wget --post-data="$post" $url >/dev/null 2>&1 && echo "$c finsihed" & 
done < list.txt

That will launch the first 10 URLs, then wait for one to finish and launch the next.

Related Question