Macos – Does Mac OS X throttle the RATE of socket creation

macosnetworkingsocketsthrottling

This may seem programming related, but this is an OS question.

I'm writing a small high performance daemon that takes thousands of connections per second. It's working fine on Linux (specifically Ubuntu 9.10 on EC2). On Mac OS X if I throw a few thousand connections at it (roughly about 16350) in a benchmark that simply opens a connection, does it's thing and closes the connection, then the benchmark program hangs for several seconds waiting for a socket to become available before continuing (or timing out in the process).

I used both Apache Bench as well as Siege (to make sure it wasn't the benchmark application).

So why/how is Mac OS X limiting the RATE at which sockets can be used, and can I stop it from doing this?

Or is there something else going on?

I know there is a file descriptor limit, but I'm not hitting that. There is no error on accepting a socket, it's simply hangs for a while after the first (roughly) 16000, waiting — I assume — for the OS to release a socket. This shouldn't happen since all prior the sockets are closed at that point. They're supposed to come available at the rate they're closed, and do on Ubuntu, but there seems to be some kind of multi (5-10?) second delay on Mac OS X.

I tried tweaking with ulimit every-which-way. Nada.

Best Answer

Mac OS X starts opening ephemeral ports at 49152. Port numbers are 16-bit unsigned integers, so there are 65535 possible ports. 65535 - 49152 = 16383. I think you've got 16K ports in TIME_WAIT.

Update: You might want to look at the following sysctl(8) variables:

net.inet.ip.portrange.lowfirst: 1023  
net.inet.ip.portrange.lowlast: 600  
net.inet.ip.portrange.first: 49152  
net.inet.ip.portrange.last: 65535  
net.inet.ip.portrange.hifirst: 49152  
net.inet.ip.portrange.hilast: 65535

I think if you set hifirst to something lower, you'll increase the number of ephemeral ports available on your system.

There might be a socket option or something to tell the stack to basically violate the TCP spec and use a nonstandard value for TIME_WAIT, but I'm not enough of a Mac OS X sockets programmer to know that.

Update 2: You probably want to use setsockopt(2) to set SO_REUSEADDR.

Related Question