Ubuntu – iptables –set-mark – Route diferent ports through different interfaces

iptablesrouteroutingUbuntu

Short story,
3 interfaces, eth0 (LAN), eth1(ADSL), eth2(4G).
eth0 -> eth1: Works
(ports 80, 443, 4070) eth0 -> eth2: Doesn't happen

This is a graphical representation of the idea:

Port 80 & 443 via eth2
ant the rest via eth1
enter image description here

Netscheme:

eth0: -ip 10.0.0.1 -net 10.0.0.0/8 -gw 10.0.0.1 (the servers own intf) 
eth1: -ip 192.168.1.74 -net 192.168.1.0/24 -gw 192.168.1.254 
eth2: -ip 192.168.1.91 -net 192.168.0.0/24 -gw 192.168.0.1

This new script reroutes 22 and 4070 to the proper table, I think.
However, after getting to that table, it doesn't get rerouted to eth2.

This script works, except for 22 and 4070!

(Port 80 is un-commented and it works but via eth1 which is wrong.)

modprobe iptable_nat
modprobe ip_conntrack

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -F PREROUTING
iptables -t nat -F
iptables -t mangle -F
iptables -F
# This next line restores any issues trying to connect to something
# if you get weird ACK packets when trying to connect (at least i did)!
iptables -t mangle -A PREROUTING -p tcp -j CONNMARK --restore-mark
ip route flush table main

iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 22 -j MARK --set-mark 1
###  iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 80 -j MARK --set-mark 1
iptables -A PREROUTING -i eth0 -t mangle -p tcp --dport 4070 -j MARK --set-mark 1

## Setup routes
# LAN
route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0
# ADSL
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth1
# 4G (Only accessible if marking packages with \x01
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth2
# Default via ADSL
## -- Does the same as ip route below? route add default gw 192.168.1.254


echo "201 eth2.out" >> /etc/iproute2/rt_tables

ip rule add fwmark 1 table eth2.out
ip route add default via 192.168.0.1 dev eth2 table eth2.out
ip route add default via 192.168.1.254 dev eth1



## Setup forwards
# From 4G to LAN
iptables -A FORWARD -i eth2 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From ADSL to LAN
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From LAN to ADSL (Default route out)
# - Note: If marked packages is sent to ADSL they will be mangled and rerouted to 4G
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

Old script:

  Ignore everything below unless you're interested in retracing my steps!!

I've created a router.sh script to setup my environment in case I do something bad.
I've got 3 ports which I want to send to a 4G connection and the rest via a landline ADSL connection.
To do this, I've instructed iptables to mangle packages on the default route and send them via my 4G interface if the –dport == 443 | 80 | 4070

However, this doesn't work; I'm still getting routed through my landline no matter what.

This is what my script looks like:

#!/bin/bash

## routing tables
# wireless = 4G via eth2
# adsl = adsl via eth1

modprobe iptable_nat
modprobe ip_conntrack

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F
ip route flush table main
ip route flush table wireless
ip route flush table adsl

## Setup routing tables
# ADSL
ip route add table adsl to 192.168.1.0/24 dev eth1
# 4G
ip route add table wireless to 192.168.0.0 dev eth2
ip rule add fwmark 0x1 table wireless

## Setup routes
# LAN
route add -net 10.0.0.0 netmask 255.0.0.0 dev eth0
# ADSL
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth1
# 4G (Only accessible if marking packages with \x01
route add -net 192.168.0.0 netmask 255.255.255.0 dev eth2
# Default via ADSL
route add default gw 192.168.1.254


## Forward ports into the LAN
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to 10.0.0.3:80


## Lets mark all packets we want for 4G forward
# HTTPS
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1
# HTTP
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 80 -j MARK --set-mark 1
# Spotify
iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 4070 -j MARK --set-mark 1

## Setup forwards
# From 4G to LAN
iptables -A FORWARD -i eth2 -o eth0 -m state --state ESTABLISHED,RELATED,NEW -j ACCEPT
# From ADSL to LAN
iptables -A FORWARD -i eth1 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
# From LAN to ADSL (Default route out)
# - Note: If marked packages is sent to ADSL they will be mangled and rerouted to 4G
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -A FORWARD -j LOG
#iptables --table nat --append POSTROUTING --out-interface eth2 --jump SNAT --to-source "192.168.1.74"
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

I've also tried to add these 3 to the bottomg of the script:

iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 80 -j SNAT --to "192.168.0.91"
iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 443 -j SNAT --to "192.168.0.91"
iptables -t nat -A POSTROUTING -o eth2 -p tcp --dport 4070 -j SNAT --to "192.168.0.91"

Also tried without success:

iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 80 -j MARK --set-mark 1

Last but not least, tried:

## Lets mark all packets we want for 4G forward
# HTTPS
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1
# HTTP
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 80 -j MARK --set-mark 1
# Spotify
iptables -A POSTROUTING -t mangle -o eth1 -p tcp --dport 4070 -j MARK --set-mark 1

The routing works, I can browse the web, listen to music and what-not, but I'm doing it through the wrong interface.
I've Googled around for a long while now and found bits and pieces to understand what I'm doing and why I'm doing it. I could do traffic shaping via tc but if it's possible via marking packages in iptables it would help me a long way.

My guess is that I'm doing the order wrong on the different rules, mainly the MASQUERADE part? or if that should even be there?

Can someone explain how to DNAT port say, tcp:80 from an external interface (either one or BOTH protocols) to a internal 10.0.0.0 address space?

Outputs:

root@Netbridge:~# route -n Kernel IP routing table Destination    

Gateway         Genmask         Flags Metric Ref    Use Iface<br>
0.0.0.0         192.168.1.254   0.0.0.0         UG    0      0        0 eth1<br>
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0<br>
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth2<br>
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

root@Netbridge:~# ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:4e  
          inet addr:10.0.0.1  Bcast:10.255.255.255  Mask:255.0.0.0

eth1      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:58  
          inet addr:192.168.1.74  Bcast:192.168.1.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 00:0c:29:7e:9e:62  
          inet addr:192.168.0.91  Bcast:192.168.0.255  Mask:255.255.255.0

Followed these instructions:
output-traffic-on-different-interfaces-based-on-destination-por
iptables-forward-specific-port-to-specific-nic

Among a few other related threads.

Best Answer

BatchyX already give some very good explanation about iptables and routing, so I will exercise my laziness and go directly to script.

It should NAT all traffic to port 80,443,22,4070 through 192.168.0.91. All the rest will NAT through 192.168.1.254.

I re-do my testing and end up following this guide. What is missing in that guide is the last 3 lines in my script. Which I found out from another port, but I lost track of that link.

It is a tested working script.

Need Default Route

One thing I did not put in the script is setting up the default route. It should be

route add default gw 192.168.1.254

When you do route -n, it should be the only default route (Dest:0.0.0.0)

0.0.0.0    192.168.1.254    0.0.0.0    UG    0    0    0    eth1

fw-router.sh

# Reset/Flush iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

#Reset/Flush/Setup IP Route (table 4)
ip route flush table 4
ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table 4 $ROUTE ; done
ip route add table 4 default via 192.168.0.1

#Mark Packet with matching D.Port
iptables -t mangle -A PREROUTING -p tcp --dport 22   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 80   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 443  -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 4070 -s 10.0.0.0/24 -j MARK --set-mark 4

#SNAT Rules
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 192.168.1.74
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 192.168.0.91

#IP Route
ip rule add fwmark 4 table 4
ip route flush cache

#IP Stack
#This is the missing part from the guide
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > $f ; done
echo 0 > /proc/sys/net/ipv4/route/flush

PS1: In short, MASQUERADE does not work (in most case, and definitely in your case) for NAT with multiple external IPs that need some kind of load balancing or need DNAT to handle incoming traffic. You need SNAT for direction control.

PS2: Pure iptables is not sufficient.

Related Solutions

OpenVPN and routing problem on OpenWRT

You have to add 10.0.0.0/24 network routed to gateway 192.168.1.2 as a static route on router1. It's not about NAT.

How to forward traffic across two Ethernet cards

Your issue is mainly the route configuration of the hosts. I assume your current setup is as follow:

enp1s0 interface has ip address 192.168.1.1/24
enp3s0 interface has ip address 192.168.176.1/24

For hosts between the two IP network to communicate, they need a dedicated entry in their routing table.

The hosts on 192.168.1.0/24 that need to access the IP camera need to know that 192.168.1.1 is the router for 192.168.176.0/24.
The hosts on 192.168.176.0/24 need a route to 192.168.1.0/24.

Now I assume that the static/DHCP configuration for the IP cameras is to route default traffic through 192.168.176.1, so they know where to send packets for the PCs. But the PCs on 192.168.1.0/24 have only one default entry the internet router. So any packet to 192.168.176.0/24 get sent there and lost.

You can either

configure your DHCP router on 192.168.1.0/24 to advertise a static route to 192.168.176.0/24 via 192.168.1.1 with the "classless static route" option
add manually 192.168.1.1 as a gateway to 192.168.176.0/24 on the PCs

You will also need to flush your iptables rules. The POSTROUTING rule will mess up the routing and the FORWARD rule is useless (unless you have a DROP policy).

iptables -t nat -F POSTROUTING
iptables -F FORWARD

Your cameras could see the PCs because they were configured with a default gateway of 192.168.176.1 and the nat POSTROUTING entry. For example if IP camera 192.168.176.10 sends a packet to PC 192.168.1.20, the packet will first be sent to 192.168.176.1 (enp3s0) the default gateway. The Ubuntu PC will forward the packet to enp1s0, rewriting the sender's address as its own, 192.168.1.1. When 192.168.1.20 replies the packet, it sends it back to the substituted address, 192.168.1.1. When the Ubuntu PC receives it, it knows it is a reply to the IP camera 192.168.176.10. So it rewrites the destination address to 192.168.176.10 and fowards it through enp3s0.

Now you don't want to mess packets with NAT, you just need IP routing. In the preceding example, the PC sees the camera IP address as 192.168.1.1, as it was substituted by the Ubuntu PC. Once you have set correct routes,

in a connection initiated by the PC to the camera, the PC will see the camera IP address as 192.168.176.10.
in a connection initiated by the camera to the PC, the PC will still see the camera IP address as 192.168.1.1 (NAT'ed address)

For simple IP devices, the second is unlikely to matter. But that could lead to buggy behaviour. As an example, if you had a managed switch and you wanted to use SNMP traps. You should delete the iptables -t nat -A POSTROUTING -j MASQUERADE rule.

One issue that will arise: Do you want your IP cameras to access the internet? If not, I would include an iptables FORWARD route to deny packets from 192.168.176.0/24 going anywhere except 192.168.1.0/24. If you wish to grant access, you will need to configure your router with a static route to 192.168.176.0/24.