I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per second through TCP while I'm only averaging 0.75 million messages per second through UDP. This seems to completely defy what I expect of the two protocols.
What's the most likely cause for the drastic difference and how can I diagnose it on Ubuntu 13.10?
#TCP RESULTS
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
87380 65536 64 10.00 1963.43 32.96 17.09 5.500 2.852
#UDP RESULTS
Socket Message Elapsed Messages CPU Service
Size Size Time Okay Errors Throughput Util Demand
bytes bytes secs # # 10^6bits/sec % SS us/KB
4194304 64 10.00 7491010 0 383.5 28.97 24.751
212992 10.00 1404941 71.9 25.03 21.381
For this test I have two test servers that are identical and directly connected via a 10G crossover cable. The NICs used in this case are Intel X520's with out-of-box configurations and connected onto a PCIe 3.0 x8 slot on the motherboard, which communicates with the CPU via a NUMA controller.
Best Answer
Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP message will result in a separate packet. In numbers: about 23 messages of size 64 byte will be combined into a single TCP packet of MTU size, while it will need 23 single packets for UDP for the same amount of data. Each of these packets means overhead with sending from the host, transmitting on the wire and receiving by the peer. And as seen in your case about 80% of the UDP packets get lost because your hardware is not fast enough to transmit and receive all these packets.
So what you can learn from this benchmark is:
As for your expectation, that UDP should be better: did you ever wonder why all the major file transfers (ftp, http,...) are done with TCP based protocols? The benchmark shows you the reason.
So why do people use UDP at all?