Bash – cURL: how to suppress strange output when redirecting

bashcurlpiperedirectionstderr

I'm trying to print just the verbose sections of a cURL request (which are sent to stderr) from the bash shell.

But when I redirect stdout like this:

curl -v http://somehost/somepage > /dev/null

Some sort of results table appears in the middle of the output to stderr:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

Followed by this near the end:

{ [data not shown]
118   592    0   592    0     0  15714      0 --:--:-- --:--:-- --:--:-- 25739

Which makes the response headers less readable.

I don't see this text when not redirecting.

Another way to see the effects:

Table doesn't appear:

curl -v http://somehost/somepage 2>&1

Table appears:

curl -v http://somehost/somepage 2>&1 | cat

1) How come this shows up only with certain types of redirects?

2) What's the neatest way to suppress it?

Thank you

Best Answer

Try this:

curl -vs -o /dev/null http://somehost/somepage 2>&1

That will suppress the progress meter, send stdout to /dev/null and redirect stderr (the -v output) to stdout.

Related Solutions

Different output when redirecting

Edit: I cannot be certain this is how mysql does it, but it could be using isatty(3) to determine whether STDOUT is a terminal or not, and modifying the output accordingly.

Edit 2: The mysql command line tool definitely uses isatty(). You can read the source code.

There are some good examples of this (although not in C) over at Stack Overflow:

To answer your question, "Why?": Because that is what the documentation says. See the reference manual:

When used interactively, query results are presented in an ASCII-table format. When used noninteractively (for example, as a filter), the result is presented in tab-separated format.

I suspect this decision was made for readability. When used interactively, mysql can assume a human is reading the output, and it is generally easier for us humans to read data that is delimited by those lines. When used non-interactively, the assumption is that another program is going to consume the output, and that tab-delimited output is easier to programmatically consume.

If you are interested in overriding this default and getting ASCII-table format when redirecting the output, you can use the --table (-t) command line option:

mysql -t -e 'select a,b from tablefoo' databasename > file

Bash – How to curl the output of another command

Use xargs.

xargs utility [argument ...]

The xargs utility reads space, tab, newline and end-of-file delimited strings from the standard input and executes utility with the strings as arguments.

^{There are more parameters and options than in this shortened form, of course.}

A general example using curl:

$ echo "http://www.google.com" | xargs curl
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.de/">here</A>.
</BODY></HTML>

In your specific case, it'd look similar to the following:

./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | xargs curl | grep myString

Best Answer

Related Solutions

Different output when redirecting

Bash – How to curl the output of another command

Related Question