Linux – Why does Linux network traffic only go through eth0

alpine-linuxlinuxnetwork-interfacenetworking

I got two NICs on the server side, eth0 ? 192.168.8.140 and eth1 ? 192.168.8.142. The client sends data to 192.168.8.142, and I expect iftop to show the traffic for eth1, but it does not. All networks go through eth0, so how can I test the two NICs?

Why does all the traffic go through eth0 instead of eth1? I expected I could get 1 Gbit/s per interface. What's wrong with my setup or configuration?

Server

ifconfig

eth0    Link encap:Ethernet  HWaddr 00:00:00:19:26:B0
        inet addr:192.168.8.140  Bcast:0.0.0.0  Mask:255.255.252.0
        inet6 addr: 0000::0000:0000:fe19:26b0/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:45287446 errors:0 dropped:123343 overruns:2989 frame:0
        TX packets:3907747 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:66881007720 (62.2 GiB)  TX bytes:261053436 (248.9 MiB)
        Memory:f7e00000-f7efffff

eth1    Link encap:Ethernet  HWaddr 00:00:00:19:26:B1
        inet addr:192.168.8.142  Bcast:0.0.0.0  Mask:255.255.255.255
        inet6 addr: 0000::0000:0000:fe19:26b1/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
        RX packets:19358 errors:0 dropped:511 overruns:0 frame:0
        TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:1000
        RX bytes:1772275 (1.6 MiB)  TX bytes:1068 (1.0 KiB)
        Memory:f7c00000-f7cfffff

Server side

# Listen for incomming from 192.168.8.142
nc -v -v -n -k -l 192.168.8.142 8000 | pv > /dev/null
Listening on [192.168.8.142] (family 0, port 8000)
Connection from 192.168.8.135 58785 received!

Client

# Send to 192.168.8.142
time yes | pv |nc -s 192.168.8.135 -4 -v -v -n 192.168.8.142 8000 >/dev/null
Connection to 192.168.8.142 8000 port [tcp/*] succeeded!

Server side

$ iftop -i eth0
interface: eth0
IP address is: 192.168.8.140

TX:             cumm:  6.34MB   peak: 2.31Mb   rates: 2.15Mb  2.18Mb  2.11Mb
RX:                    2.55GB          955Mb           874Mb   892Mb   872Mb
TOTAL:                 2.56GB          958Mb           877Mb   895Mb   874Mb

$ iftop -i eth1
interface: eth1
IP address is: 192.168.8.142

TX:             cumm:      0B   peak:     0b   rates:     0b      0b      0b
RX:                    4.51KB         3.49Kb          3.49Kb  2.93Kb  2.25Kb
TOTAL:                 4.51KB         3.49Kb          3.49Kb  2.93Kb  2.25Kb

$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:00:00:19:26:b0 brd ff:ff:ff:ff:ff:ff
$ ip link show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:00:00:19:26:b1 brd ff:ff:ff:ff:ff:ff

Best Answer

There are two possible design models for a TCP/IP network stack: a strong host model and a weak host model. You're expecting behavior that would match the strong host model. Linux is designed to use the weak host model. In general the weak host model is more common as it reduces the complexity of the routing code and thus might offer better performance. Otherwise the two host models are just different design principles: neither is inherently better than the other.

Basically, the weak host model means that outgoing traffic will be sent out the first interface listed in the routing table that matches the IP address of the destination (or selected gateway, if the destination is not reachable directly), without regard to the source IP address.

This is basically why it's generally inadvisable to use two separate physical interfaces if you need two IP addresses on the same network segment. Instead assign two IP addresses for one interface (IP aliases: e.g. eth1 = 192.168.8.142 and eth1:0 = 192.168.8.140). If you need more bandwidth than a single interface can provide, bond (or team, if applicable) two or more interfaces together, and then run both IPs on the bond/team.

By tweaking a number of sysctl settings and using the "advanced routing" functionality to set up independent routing tables for each NIC, it is possible to make Linux behave like a strong-host-model system. But that is a very special configuration, and I would recommend thinking twice before implementing it.

See the answers at Linux Source Routing, Strong End System Model / Strong Host Model? if you really need it.

Related Question