Linux – Multigateway Routing for Specific Source Port

ipiptableslinuxnetworkingrouting

I have two gateway to access internet, somehow I want to load balancing it, so far its working, but some connection or service need persistent gateway IP to be used, so the client should never change its gateway once it have connected to dest., my current implementation seems to be round-robin or whatever it is.

this is my iproute

...
...
default
    nexthop via 192.168.1.1 dev eth0 weight 1
    nexthop via 192.168.1.2 dev eth0 weight 1

now i want to fix it somehow the gateway the client will use is predetermined, for example by using source port, if the source port is even number we use gw.1 and odd number go through gw.1, can we do that using ip route?

*note that I only have one outbound interface : eth0 here.

Best Answer

Use policy routing with marking packets. I'm not sure what format of configuration it is but you should check it in your distro.

In commandline it should look like (not tested but should work)

iptables -A PREROUTING -t mangle -p tcp --dport 22 --set-mark 0x1 -j CONNMARK
echo "200 ssh" >> /etc/iproute2/rt_tables
ip rule add fwmark 1 table ssh
ip route add default dev eth0 via 192.168.1.2 table ssh

Edit: lines

echo "200 ssh" >> /etc/iproute2/rt_tables

Names routing table 200 by name "ssh". It is preserved between boots.

Related Solutions

Linux Router – How to Use Multiple Internet Providers

Here is a similar setup from one of our routers (with some irrelevant stuff snipped). Note that this handles incoming connections as well.

Note the use of variables instead of hard-coded mark numbers. So much easier to maintain! They're stored in a separate script, and sourced in. Table names are configured in /etc/iproute2/rt_tables. Interface names are set in /etc/udev/rules.d/70-persistent-net.rules.

##### fwmark ######
iptables -t mangle -F
iptables -t mangle -X

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j RETURN # if already set, we're done
iptables -t mangle -A PREROUTING -i wan      -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A PREROUTING -i comcast  -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A PREROUTING -i vz-dsl   -j MARK --set-mark $MARK_VZDSL

iptables -t mangle -A POSTROUTING -o wan     -j MARK --set-mark $MARK_CAVTEL
iptables -t mangle -A POSTROUTING -o comcast -j MARK --set-mark $MARK_COMCAST
iptables -t mangle -A POSTROUTING -o vz-dsl  -j MARK --set-mark $MARK_VZDSL
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

##### NAT ######
iptables -t nat -F
iptables -t nat -X
for local in «list of internal IP/netmask combos»; do
    iptables -t nat -A POSTROUTING -s $local -o wan     -j SNAT --to-source «IP»
    iptables -t nat -A POSTROUTING -s $local -o comcast -j SNAT --to-source «IP»
    iptables -t nat -A POSTROUTING -s $local -o vz-dsl  -j SNAT --to-source «IP»
done

# this is an example of what the incoming traffic rules look like
for extip in «list of external IPs»; do
    iptables -t nat -A PREROUTING   -p tcp -d $extip --dport «port» -j DNAT --to-destination «internal-IP»:443
done

And the rules:

ip rule flush
ip rule add from all               pref 1000  lookup main 
ip rule add from A.B.C.D/29        pref 1500  lookup comcast # these IPs are the external ranges (we have multiple IPs on each connection)
ip rule add from E.F.G.H/29        pref 1501  lookup cavtel
ip rule add from I.J.K.L/31        pref 1502  lookup vzdsl
ip rule add from M.N.O.P/31        pref 1502  lookup vzdsl # yes, you can have multiple ranges
ip rule add fwmark $MARK_COMCAST   pref 2000  lookup comcast
ip rule add fwmark $MARK_CAVTEL    pref 2001  lookup cavtel
ip rule add fwmark $MARK_VZDSL     pref 2002  lookup vzdsl
ip rule add                        pref 2500  lookup comcast # the pref order here determines the default—we default to Comcast.
ip rule add                        pref 2501  lookup cavtel
ip rule add                        pref 2502  lookup vzdsl
ip rule add                        pref 32767 lookup default

The routing tables get set up in /etc/network/interfaces, so that taking down an interface makes it switch to using a different one:

iface comcast inet static
        address A.B.C.Q
        netmask 255.255.255.248
        up ip route add table comcast default via A.B.C.R dev comcast
        down ip route flush table comcast

Note: If you're doing filtering as well (which you probably are) you'll also need to add the appropriate rules to FORWARD to ACCEPT the traffic. Especially for any incoming traffic.

Wireguard Connection Between Two LANs Behind Routers

The solution to this is pretty straightforward, however, I have not found any description on internet how to make such site-to-site VPN. Therefore, I decided to post a solution that works for me, here.

Disclaimer: This solution works for IPv4 only. Since one of my ISPs does not provide any IPv6, I did not configure for IPv6. That remains for the future improvement.

IP forwarding

First IP forwarding needs to be enabled, so nanopi will forward traffic from one interface to another. A line

net.ipv4.ip_forward=1

should be present in /etc/sysctl.conf or in some .conf file in /etc/sysctl.d/. Run sysctl -p or service procps reload (depending on linux distribution), or just reboot to make the change effective.

Wireguard

I will not go into the details how to configure wireguard. There are plenty of them on the internet. I will just write some network details used in my case, and point out two atypical configuration changes that I did in /etc/wireguard/wg0.conf. As mentioned in the question there are two sites:

192.168.0.0/24 with nanopi acting as a wireguard server with wireguard address 10.0.0.1 on wg0 interface.
192.168.1.0/24 with nanopi acting as a wireguard client with wireguard address 10.0.0.2 on wg0 interface.

Changes in /etc/wireguard/wg0.conf are:

First, an instruction is added to prevent wg-quick to setup its ip rules and routes. A line Table = off is added to [Interface] section:

Table = off

Second, add a route to direct traffic through VPN connection. Again in [Interface] section:

PostUp = ip route add 192.168.1.0/24 via 10.0.0.2 dev wg0
PreDown = ip route del 192.168.1.0/24 via 10.0.0.2 dev wg0

The lines above are from the first site, and tell the kernel to route all traffic for another network (192.168.1.0/24) to the IP connected on another side of VPN (10.0.0.2) via device wg0. On another site IP numbers in configuration are different (192.168.0.0/24, and 10.0.0.1).

Third, the second wireguard computer acting as a wireguard client has an Endpoint defined in the [Peer] section, while the first one, acting as a server, does not. That endpoint is public IP and port, where server nanopi is accessible. On the router, the server nanopi has to be exposed to the internet in order to allow incoming connections. On the site where the server wireguard computer is, the internet router shall have NAT or Port forwarding or something like that. There should be UDP on port, where wireguard connects, forwarded to the IP and port of the wireguard server IP and port. I will not show that here, because every router, has different GUI for setting that up.

Routing & DHCP

Now, when the wireguard connection works, you should be able to access nano pi on another site via VPN. While logged to 192.168.0.250 (10.0.0.1 on wg0) one should be able to ping (or login to) 10.0.0.2, and vice versa. However, other clients on both networks do not have information how to reach there through VPN. They have a default gateway where they forward all the traffic that is not for the local network. Since addresses from the other site are not local, all traffic that should go through VPN goes out to the internet via ISP routers.

One way would be to add custom static route on each device on the network. Some allow that, but many don't. Windows or linux computers have an option to add a route. On android devices that is already a problem. So, other solutions should be found, where one do not need to set a static route on each device.

If ISP router has an option to add custom routes, such a route can be added. I cannot show here how exactly that can be accomplished. In general, all addresses from another site (e.g. 192.168.0.0 with netmask 24 or 255.255.255.0 shall be redirected to 192.168.1.250).

If that is not possible, I would recommend setting up a DHCP server on nanopi and disable DHCP function on ISP router. isc-dhcp-server can be installed on practically all linux distributions, and that server can send proper routing information to all clients on the network, a long as they use DHCP. I will not go into details of DHCP server configuration. Here is an example of /etc/dhcp/dhcpd.conf:

default-lease-time 600;
max-lease-time 7200;

option subnet-mask 255.255.255.0;
option broadcast-address 192.168.0.255;
option routers 192.168.0.1;
option domain-name "localdomain";
option domain-name-servers 192.168.0.3;

option rfc3442-classless-static-routes code 121 = array of integer 8;
option ms-classless-static-routes code 249 = array of integer 8;

option rfc3442-classless-static-routes 24, 192, 168, 1, 192, 168, 0, 250, 0, 192, 168, 0, 1;
option ms-classless-static-routes 24, 192, 168, 1, 192, 168, 0, 250, 0, 192, 168, 0, 1;

subnet 192.168.0.0 netmask 255.255.255.0 {
    range 192.168.0.150 192.168.0.229;

    default-lease-time 86400;
    max-lease-time 172800;
}

host staticip {
    default-lease-time 86400;
    max-lease-time 172800;

    hardware ethernet aa:bb:cc:dd:ee:ff;
    fixed-address 192.168.0.4;
}

Everything in dhcp configuration is pretty standard. DHCP is assigning addresses from 192.168.0.150 to 192.168.0.229. Change that to your preferences.

I should point out the lines option rfc3442... and option ms-classless.... They define options for DHCP to send route information. One is for clients following RFC 3442, and another for Microsoft clients, which use another option. The numbers 24, 192, 168, 1, 192, 168, 0, 250, 0, 192, 168, 0, 1 have the following meaning:

First static route:
- 24 - netmask size
- 192, 168, 1 - prefix for the netmask defined above
- 192, 168, 0, 250 - gateway for the network defined above
Second static route:
- 0 - netmask size (default gateway)
- no network since the mask above is 0
- 192, 168, 0, 1 - gateway for the network defined above

Default route has to be provided here, too, because default route should get ignored on client if this information is received via DHCP.

At the end of dchp config file there is an example how to make a DHCP reservation for clients that will have static IP addresses. Enter you hardware ethernet (MAC) address and desired IP.

Firewalls

Now clients from one network should be able access clients on another network via wireguard VPN, provided firewalls will let the traffic through.

If there is a firewall on your VPN computer (e.g. nanopi) it may not allow traffic to enter. Make firewall rules according to your preferences and security requirements of your environment. You can open each individual port that you need to go through on a VPN computer, or you can open all, if it is in a trusted environment. One way to do that is with using firewall-cmd commands. Here is an example:

# set "trusted" as a default zone
firewall-cmd --set-default-zone=trusted

# see in which zones the interfaces are
firewall-cmd --get-active-zones

# you may have to remove eth0 from the zone where it belongs to (public in this case)
firewall-cmd --permanent --zone=public --remove-interface=eth0
# add eth0 to trusted zone
firewall-cmd --permanent --zone=trusted --add-interface=eth0

# you may have to remove wg0 from the zone where it belongs to (public in this case)
firewall-cmd --permanent --zone=public --remove-interface=wg0
# add wg0 to trusted zone
firewall-cmd --permanent --zone=trusted --add-interface=wg0

# reload the new config
firewall-cmd --reload

Then there are firewalls on each device that may stop the incoming traffic. Typically, Windows firewall allows some connections from "local network" only. Another site is not on local network, so server will block the connections coming through VPN. You have to add another network (e.g. 192.168.1.0/24) for each rule that is blocking a particular connection. For instance, for Windows share you have to change all incoming rules for ports 135-139, and 445 on computer sharing. Similarly, you have to change outgoing rules on a computer accessing the share.

I would certainly recommend some time to fine tune the firewalls instead of just shutting them off.

I hope the instructions are clear enough and will help to setup a site-to-site VPN or at least give an idea how to do it. Some knowledge in networking and administration is required.