Send traffic to self over physical network on Ubuntu

The Linux kernel version 2.6.35 introduces a new configuration option CONFIG_NETFILTER_XT_TARGET_TEE:

This option adds a "TEE" target with which a packet can be cloned and this clone be rerouted to another nexthop.

Iptables supports the -j TEE target since 1.4.8.

Earlier support was through the xtables addons, which include both kernel modules and userland tools. You may still prefer this option if you prefer to stick with your distribution's kernel and it's too old to have TEE.

There's a tutorial by bjou (written before the feature was included in the official kernel and iptables).

Here is a similar setup from one of our routers (with some irrelevant stuff snipped). Note that this handles incoming connections as well.

Note the use of variables instead of hard-coded mark numbers. So much easier to maintain! They're stored in a separate script, and sourced in. Table names are configured in /etc/iproute2/rt_tables. Interface names are set in /etc/udev/rules.d/70-persistent-net.rules.

##### fwmark ######
iptables -t mangle -F
iptables -t mangle -X

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN # if already set, we're done
iptables -t mangle -A PREROUTING -i wan      -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A PREROUTING -i comcast  -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A PREROUTING -i vz-dsl   -j MARK --set-mark $MARK_VZDSL

iptables -t mangle -A POSTROUTING -o wan     -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A POSTROUTING -o comcast -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A POSTROUTING -o vz-dsl  -j MARK --set-mark $MARK_VZDSL
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

##### NAT ######
iptables -t nat -F
iptables -t nat -X
for local in «list of internal IP/netmask combos»; do
    iptables -t nat -A POSTROUTING -s $local -o wan     -j SNAT --to-source «IP»
    iptables -t nat -A POSTROUTING -s $local -o comcast -j SNAT --to-source «IP»
    iptables -t nat -A POSTROUTING -s $local -o vz-dsl  -j SNAT --to-source «IP»
done

# this is an example of what the incoming traffic rules look like
for extip in «list of external IPs»; do
    iptables -t nat -A PREROUTING   -p tcp -d $extip --dport «port» -j DNAT --to-destination «internal-IP»:443
done

And the rules:

ip rule flush
ip rule add from all               pref 1000  lookup main 
ip rule add from A.B.C.D/29        pref 1500  lookup comcast # these IPs are the external ranges (we have multiple IPs on each connection)
ip rule add from E.F.G.H/29        pref 1501  lookup cavtel
ip rule add from I.J.K.L/31        pref 1502  lookup vzdsl
ip rule add from M.N.O.P/31        pref 1502  lookup vzdsl # yes, you can have multiple ranges
ip rule add fwmark $MARK_COMCAST   pref 2000  lookup comcast
ip rule add fwmark $MARK_CAVTEL    pref 2001  lookup cavtel
ip rule add fwmark $MARK_VZDSL     pref 2002  lookup vzdsl
ip rule add                        pref 2500  lookup comcast # the pref order here determines the default—we default to Comcast.
ip rule add                        pref 2501  lookup cavtel
ip rule add                        pref 2502  lookup vzdsl
ip rule add                        pref 32767 lookup default

The routing tables get set up in /etc/network/interfaces, so that taking down an interface makes it switch to using a different one:

iface comcast inet static
        address A.B.C.Q
        netmask 255.255.255.248
        up ip route add table comcast default via A.B.C.R dev comcast
        down ip route flush table comcast

Note: If you're doing filtering as well (which you probably are) you'll also need to add the appropriate rules to FORWARD to ACCEPT the traffic. Especially for any incoming traffic.