Linux – How does iptables MASQUERADE work on the incoming side

OK, I figured it out. What I had to do was add the internal subnet route to each route table then set rules to control what interface traffic routes to/from. Then in iptables marking packets with the mangle table was not needed, just the typical forward and nat rules.

ip route add 136.2.178.32/28 dev eth4 table 1
ip route add 10.6.0.0/20 dev eth3 table 1
ip route add default via 136.2.178.33 src 136.2.178.37 table 1
ip rule add iif eth4 table 1
ip rule add from 10.6.0.0/20 table 1

ip route add 136.2.217.96/28 dev eth2 table 2
ip route add 10.1.0.0/16 dev eth1 table 2
ip route add default via 136.2.217.113 src 136.2.217.124 table 2
ip rule add iif eth2 table 2
ip rule add from 10.1.0.0/16 table 2

iptables -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j     ACCEPT
iptables -A FORWARD -i eth1 -o eth2 -m state --state NEW -j LOG --log-level debug
iptables -A FORWARD -i eth1 -o eth2 -j ACCEPT
iptables -A FORWARD -i eth4 -o eth3 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth3 -o eth4 -m state --state NEW -j LOG --log-level debug
iptables -A FORWARD -i eth3 -o eth4 -j ACCEPT
iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth4 -j MASQUERADE

For a NAT to work properly both the packets from client to server and the packets from server to client must pass through the NAT.

Note that the NAT table in iptables is only used for the first packet of a connection. Later packets related to the connection are processed using the internal mapping tables established when the first packet was translated.

iptables -t nat -A PREROUTING -i br-lan -s 192.168.1.0/24 -d 82.120.11.22/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.200

With just this rule in place the following happens.

• The client creates the initial packet (tcp syn) and addresses it to the public IP. The client expects to get a response to this packet with the source ip/port and destination ip/port swapped.
• Since the client has no specific entries in its routing table it sends it to its default gateway. The default gateway is the NAT box.
• The NAT box receives the intial packet, modifies the destination IP, establishes a mapping table entry, looks up the new destination in its routing table and sends the packets to the server. The source address remains unchanged.
• The Server receives the initial packet and crafts a response (syn-ack). In the response the source IP/port is swapped with the destination IP/port. Since the source IP of the incoming packet was unchanged the destination IP of the reply is the IP of the client.
• The Server looks up the IP in its routing table and sends the packet back to the client.
• The client rejects the packet because the source address doesn't match what it expects.

iptables -t nat -A POSTROUTING -o br-lan -s 192.168.1.0/24 -d 192.168.1.200/32 -p tcp -m tcp --dport 80 -j SNAT --to-source 192.168.1.1

Once we add this rule the sequence of events changes.

• The client creates the initial packet (tcp syn) and addresses it to the public IP. The client expects to get a response to this packet with the source ip/port and destination ip/port swapped.
• Since the client has no specific entries in its routing tables it sends it to its default gateway. The default gateway is the NAT box.
• The NAT box receives the intial packet, following the entries in the NAT table it modifies the destination IP, source IP and possiblly source port (source port is only modified if needed to disambiguate), establishes a mapping table entry, looks up the new destination in its routing table and sends the packets to the server.
• The Server receives the initial packet and crafts a response (syn-ack). In the response the source IP/port is swapped with the destination IP/port. Since the source IP of the incoming packet was modified by the NAT box the destination IP of the packet is the IP of the NAT box.
• The Server looks up the IP in its routing table and sends the packet back to the NAT box.
• The NAT box looks up the packet's details (source IP, source port, destination IP, destination port) in its NAT mapping tables and performs a reverse translation. This changes the source IP to the public IP, the source port to 80, the destination IP to the client's IP and the destination port back to whatever source port the client used.
• The NAT box looks up the new destination IP in its routing table and sends the packet back to the client.
• The client accepts the packet.
• Communication continues with the NAT translating packets back and forth.