Stop wget reusing existing connection

networkingwget

so I am trying to wget a specific webpage using this command in bash scripting:

wget --no-cookies --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" -O $2/content.html $1

And the result is that I get the bot page from the website because wget is reusing the existing connection (I think). This command was working before I spam tested and now my server is getting a bot test redirect from the site (can't use this).

--2017-12-12 19:16:42--  https://www.kayak.co.uk/h/bots/human-redirect.vtl?url=%2Fflights%2FDUB-LAX%2F2018-06-04%2F2018-06-25%2F2adults%3Fsort%3Dbestflight_a
Reusing existing connection to [www.kayak.co.uk]:443.
HTTP request sent, awaiting response... 200 OK

My question is: is there anyway to stop wget from using the existing connection and reconnect the site to download each time?

Best Answer

I know this is an old issue, but perhaps this will help others who come across it as I have.

To disable the "keep-alive" feature, use the --no-http-keep-alive argument.

From the man page:

Turn off the "keep-alive" feature for HTTP downloads. Normally, Wget asks the server to keep the connection open so that, when you download more than one document from the same server, they get transferred over the same TCP connection. This saves time and at the same time reduces the load on the server.

Using this argument is typically needed in cases where a new, clean request is necessary. Although not strictly related, the --no-cache and --no-cookies arguments might also be relevant in cases where the --no-http-keep-alive argument is used.

So the OP's command would probably be:

wget --no-http-keep-alive --no-cache --no-cookies --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" -O $2/content.html $1

Related Solutions

Bash – How you stop ‘wget’ after it gets a 404

If you're happy with a loop:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    wget "$url" || break
done

That will run wget for each URL in your expansion until it fails, and then break out of the loop.

If you want two failures in a row it gets a bit more complicated:

for url in 'http://www.iqandreas.com/sample-images/100-100-color/'{90..110}'.jpg'
do
    if wget "$url"
    then
        failed=
    elif [ "$failed" ]
    then
        break
    else
        failed=yes
    fi
done

You can shrink that a little with && and || instead of if, but it gets pretty ugly.

I don't believe wget has anything built in to do that.

Wget Directory Options

(untested) The following needs some tunning, is just a general idea:

### get level1
wget -r -l  website.com/      

#### for each html file otained,
for a in $(find website.com -name '*.html' )
do 
  ### get level 2 but prefix it with the base name
  b=$(basename $a)
  wget -P $b -r -l 1 -A pdf http://$a 
done

probably the find will need some tunnig
perhaps add something like mv $b/website.com/files FINAL/$b to reduce the levels

Best Answer

Related Solutions

Bash – How you stop ‘wget’ after it gets a 404

Wget Directory Options

Related Question