networking – Linux Does Not Reply to ARP Request Messages if Requested IP Address is Associated with Another (Disabled) Interface

arplinux-kernelnetworking

I have a PC(kernel 3.2.0-23-generic) which has 192.168.1.2/24 configured to eth0 interface and also uses 192.168.1.1 and 192.168.1.2 addresses for tun0 interface:

root@T42:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:16:41:54:01:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 scope global eth0
    inet6 fe80::216:41ff:fe54:193/64 scope link
       valid_lft forever preferred_lft forever
3: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: irda0: <NOARP> mtu 2048 qdisc noop state DOWN qlen 8
    link/irda 00:00:00:00 brd ff:ff:ff:ff
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:13:ce:8b:99:3e brd ff:ff:ff:ff:ff:ff
    inet 10.30.51.53/24 brd 10.30.51.255 scope global eth1
    inet6 fe80::213:ceff:fe8b:993e/64 scope link
       valid_lft forever preferred_lft forever
6: tun0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc pfifo_fast state DOWN qlen 100
    link/none
    inet 192.168.1.1 peer 192.168.1.2/32 scope global tun0
root@T42:~# ip route show dev eth0
192.168.1.0/24  proto kernel  scope link  src 192.168.1.2 
root@T42:~#

As seen above, tun0 is administratively disabled(ip link set dev tun0 down). Now when I receive ARP requests for 192.168.1.2, the PC does not reply to those requests:

root@T42:~# tcpdump -nei eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:30:34.875427 00:1a:e2:ae:cb:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.2 tell 192.168.1.1, length 46
15:30:36.875268 00:1a:e2:ae:cb:b7 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.2 tell 192.168.1.1, length 46
15:30:39.138651 00:1a:e2:ae:cb:b7 > 00:1a:e2:ae:cb:b7, ethertype Loopback (0x9000), length 60:
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@T42:~#

Only after I delete the tun0 interface(ip link del dev tun0) the PC will reply to ARP request for 192.168.1.2 on eth0 interface.

Routing table looks exactly alike before and after ip link del dev tun0:

root@T42:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.30.51.254    0.0.0.0         UG        0 0          0 eth1
10.30.51.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.1.0     192.168.1.2     255.255.255.0   UG        0 0          0 eth0
root@T42:~# ip link del dev tun0
root@T42:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.30.51.254    0.0.0.0         UG        0 0          0 eth1
10.30.51.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.1.0     192.168.1.2     255.255.255.0   UG        0 0          0 eth0
root@T42:~#

Routing entry below is removed already with ip link set dev tun0 down command:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.2     0.0.0.0         255.255.255.255 UH        0 0          0 tun0

However, while routing tables are exactly alike before and after the ip link del dev tun0 command, the actual routing decisions kernel will make are not:

T42:~# ip route get 192.168.1.1
local 192.168.1.1 dev lo  src 192.168.1.1 
    cache <local> 
T42:~# ip link del dev tun0
T42:~# ip route get 192.168.1.1
192.168.1.1 dev eth0  src 192.168.1.2 
    cache  ipid 0x8390
T42:~#

Is this an expected behavior? Why does kernel ignore the routing table?

Best Answer

Your routing table isn't being ignored, exactly. It's being overruled by a higher-priority routing table.

What's Going On

The routing table you see when you type ip route show isn't the only routing table the kernel uses. In fact, there are three routing tables by default, and they are searched in the order shown by the ip rule command:

# ip rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The table you're most familiar with is main, but the highest-priority routing table is local. This table is managed by the kernel to keep track of local and broadcast routes: in other words, the local table tells the kernel how to route to the addresses of its own interfaces. It looks something like this:

# ip route show table local
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1
broadcast 192.168.1.0 dev eth0  proto kernel  scope link  src 192.168.1.2
local 192.168.1.1 dev tun0  proto kernel  scope host  src 192.168.1.1
local 192.168.1.2 dev eth0  proto kernel  scope host  src 192.168.1.2
broadcast 192.168.1.255 dev eth0  proto kernel  scope link  src 192.168.1.2

Check out that line referencing tun0. That's what's causing your strange results from route get. It says 192.168.1.1 is a local address, which means if we want to send an ARP reply to 192.168.1.1, it's easy; we send it to ourself. And since we found a route in the local table, we stop searching for a route, and don't bother checking the main or default tables.

Why multiple tables?

At a minimum, it's nice to be able to type ip route and not see all those "obvious" routes cluttering the display (try typing route print on a Windows machine). It can also serve as some minimal protection against misconfiguration: even if the main routing table has gotten mixed up, the kernel still knows how to talk to itself.

(Why keep local routes in the first place? So the kernel can use the same lookup code for local addresses as it does for everything else. It makes things simpler internally.)

There are other interesting things you can do with this multiple-table scheme. In particular, you can add your own tables, and specify rules for when they are searched. This is called "policy routing", and if you've ever wanted to route a packet based on its source address, this is how to do it in Linux.

If you're doing especially tricky or experimental things, you can add or remove local routes yourself by specifying table local in the ip route command. Unless you know what you're doing, though, you're likely to confuse the kernel. And of course, the kernel will still continue to add and remove its own routes, so you have to watch to make sure yours don't get overwritten.

Finally, if you want to see all of the routing tables at once:

# ip route show table all

For more info, check out the ip-rule(8) man page or the iproute2 docs. You might also try the Advanced Routing and Traffic Control HOWTO for some examples of what you can do.

Related Question