I have a PC with two internet connections. The first one is an Ethernet interface to connect to my local network and access internet via my router. The second one use a GSM modem to connect with ppp.
I would like to be able to use each connection independently when specifying the interface, e.g.:
ping -I eth0 www.google.com
ping -I ppp0 www.google.com
And to be able to use eth0 by default if no interface is selected and the connection via eth0 works but ppp0 if the connection via eth0 does not work.
I read the this article and created the following rules:
# Main table
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100
ip route add 10.64.64.64 dev ppp0 src 10.123.122.101
ip route add default via 10.0.0.1
# Specific tables
ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100 table eth0
ip route add default via 10.0.0.1 table eth0
ip route add 10.64.64.64 dev ppp0 src 10.123.122.101 table ppp0
ip route add default via 10.64.64.64 table ppp0
# Rules
ip rule add from 10.0.0.100 table eth0
ip rule add from 10.123.122.101 table ppp0
At first it seems to work. However the eth0 table seems to be never used. I expected the following to happen:
ping -I eth0 www.google.ch # Use default gateway in table eth0
ping -I ppp0 www.google.ch # Use default gateway in table ppp0
ping www.google.ch # Use default gateway (main)
However if I remove the default gateway (main) the eth0 interface does not work at all. There is obviously something I don't understand. Can you explain me how to achieve what I want?
EDIT according to @derobert answer:
I tested the following configuration but still get the same error (ok for ppp0 but not working for eth0).
Routes:
# ip rule list
0: from all lookup local
1500: from 10.0.0.100 lookup eth0
1501: from 10.123.122.101 lookup ppp0
2000: from all fwmark 0x1 lookup eth0
2001: from all fwmark 0x2 lookup ppp0
32766: from all lookup main
32767: from all lookup default
# ip route list table eth0
10.0.0.0/24 dev eth0 src 10.0.0.100
default via 10.0.0.1 dev eth0
# ip route list table ppp0
10.64.64.64 dev ppp0 src 10.123.122.101
default via 10.64.64.64 dev ppp0
# ip route list table main
10.64.64.64 dev ppp0 src 10.123.122.101
192.168.1.0/24 dev eth1 src 192.168.1.1
10.0.0.0/24 dev eth0 src 10.0.0.100
Firewall:
# iptables -L
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
CONNMARK all -- anywhere anywhere CONNMARK restore
RETURN all -- anywhere anywhere mark match !0x0
MARK all -- anywhere anywhere MARK set 0x1
MARK all -- anywhere anywhere MARK set 0x2
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MARK all -- anywhere anywhere MARK set 0x1
MARK all -- anywhere anywhere MARK set 0x2
CONNMARK all -- anywhere anywhere CONNMARK save
# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- 10.0.0.1 anywhere to:10.0.0.100
SNAT all -- 10.64.64.64 anywhere to:10.123.122.101
I removed all the other firewall rules to be sure they do not interfere. I get this results:
# ip route get 8.2.1.1 from 10.0.0.100
8.2.1.1 from 10.0.0.100 via 10.0.0.1 dev eth0
# ip route get 8.2.1.1 from 10.123.122.101
8.2.1.1 from 10.123.122.101 via 10.64.64.64 dev ppp0
# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=45 time=350.108 ms
64 bytes from 8.8.8.8: seq=1 ttl=45 time=349.768 ms
64 bytes from 8.8.8.8: seq=2 ttl=45 time=329.671 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 329.671/343.182/350.108 ms
128# ping -I eth0 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
The rules seems to be OK but obviously there is another problem. I don't really understand the role of the SNAT here. I added the rules to reflect the configuration of the answer be they might be wrong.
Best Answer
I have a slightly more complicated configuration, which also includes NAT and dynamic (internal) routing on the machine.
There are several parts to it. Let's start with the rules:
(The tables are given names in
/etc/iproute2/rt_tables)As you can see, we have three ISPs at the moment, and we a couple static IP addresses on each. Rules 1500–1502 send the traffic from those source IP addresses out the appropriate interface. Rules 2000–2002 send traffic with a given firewall mark (we'll get to that) out the appropriate interface. Rules 2500–2502 give the preference order of our ISPs, for traffic that hasn't been assigned to an ISP already. If one goes down, you remove the rule for it, and the next one on the list is used.
Each routing table is fairly trivial:
(Local and main have a lot more stuff, but that's directly connected and internal routes, respectively. Table default is empty).
Next, its important that once a connection is assigned to an ISP, it stays there (including incoming connections). Trying to move it is impossible, since we've got different IP addresses on each interface, and our ISPs actually have reverse-path filtering. We keep them on the same ISP with firewall rules.
Remember that SNAT is done after routing, so the
ip rulepolicies will not help. You have to use something else (maybe this is your problem?)This is from the shell script that sets it up; the
$MARK_…variables match the marks you see in the rules above. Fairly simple: restore the connection mark (remember, marks are per-packet); if there is a mark now (from the connection mark), we're done. Otherwise, set the mark to the one based on the interface involved.Note it restores the mark before routing, and only saves it afterwards. And the mark is set to the outgoing interface regardless (which is arguable).
Finally, there are the actual NAT rules. We have a several local prefixes, this code is run in a for loop with
$localset to each:(Note: There are a lot more rules, including some DNAT for DMZ servers, etc... I think I've copied all the relevant ones.)