What’s the difference between using netcat (nc) and curl for HTTP requests

.nccurlhttpnetcat

I'm using curl to request a specific URL and getting 200 OK response:

curl -v www.youtypeitwepostit.com
* About to connect() to www.youtypeitwepostit.com port 80 (#0)
*   Trying 54.197.246.21...
* Connected to www.youtypeitwepostit.com (54.197.246.21) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.youtypeitwepostit.com
> Accept: */*
>
< HTTP/1.1 200 OK
...

If I save headers to file as:

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: www.youtypeitwepostit.com
Accept: */*

and try to execute nc command (netcat):

nc www.youtypeitwepostit.com 80 < file
HTTP/1.1 505 HTTP Version Not Supported
Connection: close
Server: Cowboy
Date: Wed, 02 Nov 2016 04:08:34 GMT
Content-Length: 0

I'm getting another response. What's the difference and how can I get 200 OK using nc?

I tried with different versions of HTTP in request header, tried to type request manually to avoid wrong CRLFs, tried to exclude optional headers. The results are similar.

Best Answer

The relevant RFC, Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing contains the answer to your question: that each line of a HTTP request should end with CR/LF.

The grammar for the HTTP Message Format specifies that each header line should end with a Carriage Return character (0x0d in ASCII) followed by a line feed character (0x0a):

 HTTP-message   = start-line
                  *( header-field CRLF )
                  CRLF
                  [ message-body ]

This is expressed more clearly in the description of the Request Line:

A request-line begins with a method token, followed by a single space (SP), the request-target, another single space (SP), the protocol version, and ends with CRLF.

 request-line   = method SP request-target SP HTTP-version CRLF

Since curl is a specifically developed for HTTP requests it already uses the appropriate line-endings when making HTTP requests. However, netcat is a more general purpose program. As a Unix utility, it uses line-feed characters for line endings by default, thus requiring the user to ensure that lines are terminated correctly.

You can use the unix2dos utility to convert the file containing the request headers to use Carriage Return / Line Feed endings.

If you want to type the HTTP request by hand and have a recent version of nc, you should use its -C option to use CRLF for line endings:

nc -C www.youtypeitwepostit.com 80

By the way, it’s worth noting that most popular Internet protocols (e.g., SMTP) use CR/LF line endings.

Note that some web servers (e.g. Apache) are more forgiving and will accept request lines that are terminated only with a Line Feed character. The HTTP specification allows for this, as mentioned in the Message Parsing Robustness section:

Although the line terminator for the start-line and header fields is the sequence CRLF, a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR.

Related Question