Mac – How to run ALL HTTP requests through Charles Web Debugging Proxy – including command line ones

debughttpmacnetworkingPROXY

I'm using the Charles Web Debugging Proxy software to debug HTTP requests. It works great with my desktop browsers, Chrome and Firefox, and it even sees HTTP requests that other programs make.

When I run Charles and check the network config, I think I understand how it works – it just sets up a proxy for all HTTP and HTTPS requests and then listens for these on port 8888:

However the item I can't figure out is that I don't see any requests that I initiate at the Terminal, such as wget, curl, or the elinks browser.

I know that I can specify the proxy with curl and wget by using 127.0.0.1:8888, but I don't understand if the network interface is set up with a proxy in the configuration why I would need to manually specify the proxy for them.

Also I can't seem to get BlueCrab (website copier) to show up in Charles either – and I don't see a proxy setting for it – although I believe it is using an XWindow wrapper or something (so it's not really a native Cocoa / Carbon app):

How can I get all HTTP requests on my system to run through Charles?

Clarification

My question is about the system fundamentals of why curl and wget wouldn't use a proxy when the network interface was set up to use one more so than asking about the correct syntax for curl, wget, etc.

Best Answer

VPN would create a new network device, you can see it in ifconfig command, and then route all system network to that device, you could see the route use route command.

But HTTP Proxy (in this case, the Charles) is different, it just open a port, to use it, you must specify your application setting to use that port for HTTP stuffs. and as Kevin Reid's answer, curl, wget etc don't read OS X's system wide settings.

If your proxy is SOCKS (Charles supports both HTTP and SOCKS), you could use ProxyChains or tsocks for application doesn't support proxy setting.

e.g.:

$ proxychains git clone https://github.com/rofl0r/proxychains-ng

Related Solutions

Windows – Proxy settings for HTTP request using libcurl

See libcurl-tutorial (3), section Environment Variables:

libcurl automatically checks and uses a set of environment
variables to know what proxies to use for certain
protocols. The names of the variables are following an
ancient de facto standard and are built up
as "[protocol]_proxy" (note the lower casing). Which makes
the variable 'http_proxy' checked for a name of a proxy to
use when the input URL is HTTP. Following the same rule, the
variable named 'ftp_proxy' is checked for FTP URLs. Again,
the proxies are always HTTP proxies, the different names of
the variables simply allows different HTTP proxies to be
used.

The proxy environment variable contents should be in the
format "[protocol://][user:password@]machine[:port]".

So you have to enter the proxy in the environment variable settings, not the network settings.

How to cause a squid proxy to log all requests

It is logging all requests.

TCP_MISS means that it did not load the response from a cache. See Squid Log Files for a listing of all tag types.

These tags are optional and describe whether the response was loaded from cache, network, or otherwise:

HIT The response object delivered was the local cache object.

MEM Additional tag indicating the response object came from memory cache, avoiding disk accesses. Only seen on HIT responses.

MISS The response object delivered was the network response object.

DENIED The request was denied by access controls.

NOFETCH A ICP specific type. Indicating service is alive, but not to be used for this request. Sent during -Y startup, or during frequent failures, a cache in hit only mode will return either UDP_HIT or UDP_MISS_NOFETCH. Neighbours will thus only fetch hits.

Refresh a page with a log of static images in a browser that is going through the proxy and you will likely see a lot of TCP_IMS_HIT which means the request was on the HTTP port (usually 3128), the client sent a revalidation (conditional) request and the response object delivered was the local cache object.

Best Answer

Related Solutions

Windows – Proxy settings for HTTP request using libcurl

How to cause a squid proxy to log all requests

Related Question