Restoring lost network routes after loss of network

ipnetworkingroute

I have a server in a locked-down environment with no egress to the internet, and 2 interfaces: a physical eth0, and a vlan iface eth0.101

/etc/network/interfaces contains a post-up command to enable a route to a specific net block via the vlan iface, like so:

post-up ip route add 10.1.0.0/24 via 10.1.2.1 dev eth0.101

During switch failover testing, we noticed that the route was lost (RTNETLINK answers: Network is unreachable.) which makes sense. However, once the network came back online, the route was not added to the interface again.

I understand why – the interface didn't go down, it just lost access to that net.

How can I configure an interface to restore routes to networks that where lost, but, to quote the old song, have now been found?

We use Debian 9 and have a service definition iface@eth0.service for each interface, which uses ifup commands to bring the device up / down. But again, the device, and the link to the switch, never faltered. I mention this in case extra systemd options can be leveraged.

Best Answer

A routing table will make your route permanent (to avoid adding it again/manually after a switch failover); First, create a named routing table. As an example, we could use "mgmt".

echo '200 mgmt' >> /etc/iproute2/rt_tables

Just for an extended detail about the solution, above, the kernel supports many routing tables and refers to these by unique integers numbered 0-255. A name, mgmt, is also defined for the table. Below, a look at a default /etc/iproute2/rt_tables follows, showing that some numbers are reserved. The choice in this answer of 200 is arbitrary; one might use any number that is not already in use, 1-252.

# reserved values
255     local
0       unspec

Second, edit your post-up rule (under /etc/network/interfaces) like this

  post-up ip route add 10.1.0.0/24 dev eth0.101 table mgmt
  post-up ip route add default via 10.1.2.1 dev eth0.101 table mgmt
  post-up ip rule add from 10.1.0.0/24 table mgmt
  post-up ip rule add to 10.1.0.0/24 table mgmt

Alternatively an other solution could be a background bash script checking for the route existence and adding it back if it's missing, the script could check the result of ip route add 10.1.0.0/24 via 10.1.2.1 dev eth0.101 the script could be setup in a loop or a cron

ip route add 10.1.0.0/24 via 10.1.2.1 dev eth0.101
if [ $? -eq 0 ]; then
    echo "Route added again"
    sleep 10;
    command-to-call-the-script-again
else
    echo "Route exists"
    sleep 10;
    command-to-call-the-script-again
fi

Source: what is the best way to add a permanent route?

Related Question