Debian – How to curl full web page content

curldebianlinuxUbuntu

I want to download the full HTML source code of a X web page, however curling the X link will return partial HTML source code as that web page requires scrolling to see further content. It seems that curl doesn't go past "scroll down".

So far, I can only do this manually:
1) Go to the desired website
2) Execute the following command in browser's console to auto-scroll (load every object):

var scroll = setInterval(function(){ window.scrollBy(0,1000); }, 2000);

3) Copy the full HTML source code from inspect element

So the question is, how can I run curl command so it scrapes full web page content (scrolls until it loads all objects) before outputing at terminal to achieve same result as the abovementioned steps? If not with curl, maybe wget?

Best Answer

curl isn't a full-fledged browser and to the best of my knowledge does not support executing JavaScript. It uses HTTP/FTP to fetch files; that is all. If you want to do testing of functionality which depends upon scripting or other tooling which a bare HTTP request is incapable of touching upon, you will need to look into a more in-depth test suite such as Selenium.

Related Solutions

curl – Automating Web Requests Using curl

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when trying to figure out what to do with curl. I've written bash scripts automating various curl invocations and use pipes and temporary files (courtesy of mktemp) to do some limited processing on the webpage, but if have to do a lot of webpage processing I usually switch to Perl with LibWWW.

Curl Monitoring – Health Check of Web Page Using Curl

What about -sSf? From the man pages:

  -s/--silent
     Silent or quiet mode. Do not show progress meter or error messages.  
     Makes Curl mute.

  -S/--show-error
     When used with -s it makes curl show an error message if it fails.

  -f/--fail
     (HTTP)  Fail silently (no output at all) on server errors. This is mostly
     done to better enable scripts etc to better deal with failed attempts. In
     normal  cases  when a HTTP server fails to deliver a document, it returns
     an HTML document stating so (which often also describes  why  and  more).
     This flag will prevent curl from outputting that and return error 22.

     This method is not fail-safe and there are occasions where non-successful
     response codes will  slip  through,  especially  when  authentication  is
     involved (response codes 401 and 407).

For example:

curl -sSf http://example.org > /dev/null

Best Answer

Related Solutions

curl – Automating Web Requests Using curl

Curl Monitoring – Health Check of Web Page Using Curl

Related Question