Ubuntu – Iptables DNAT not working for some destination interface

iptablesnetworkingUbuntu

I want http traffic to be forwarded to 192.168.99.100:32150.

I use:

sudo iptables -A OUTPUT -t nat -o lo -p tcp --dport 80 -j DNAT --to-destination 192.168.99.100:32150

I expect that telnet 127.0.0.1 80 connects to 192.168.99.100:32150.
For some reason it just hangs forever (TRACE logs below). I know that 192.168.99.100:32150 is listening because telnet 192.168.99.100 32150 does connect successfully.

Surprisingly, it does work as expected when the DNAT destination endpoint is 192.168.1.3:32150 (wlp3s0f0 interface) instead of 192.168.99.100:32150 (vboxnet0 interface). It seems that for some reason DNAT does not work if the destination interface is vboxnet0 but it does work for wlp3s0f0.

What could possibly cause such behaviour? Is my rule incorrect?

uname -a:

Linux xxx-X550LC 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

ifconfig:

conjureup0 Link encap:Ethernet  HWaddr de:6c:af:10:43:16  
          inet addr:10.99.0.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::dc6c:afff:fe10:4316/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:373 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:67696 (67.6 KB)

docker0   Link encap:Ethernet  HWaddr 02:42:4f:04:7a:6c  
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:4fff:fe04:7a6c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:198 errors:0 dropped:0 overruns:0 frame:0
          TX packets:570 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:52783 (52.7 KB)  TX bytes:90304 (90.3 KB)

enp2s0f1  Link encap:Ethernet  HWaddr bc:ee:7b:19:f6:ff  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:16471 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:103617080 (103.6 MB)  TX bytes:103617080 (103.6 MB)

vboxnet0  Link encap:Ethernet  HWaddr 0a:00:27:00:00:00  
          inet addr:192.168.99.1  Bcast:192.168.99.255  Mask:255.255.255.0
          inet6 addr: fe80::800:27ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:270 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:50604 (50.6 KB)

veth86607c2 Link encap:Ethernet  HWaddr 76:05:f3:47:87:0f  
          inet6 addr: fe80::7405:f3ff:fe47:870f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:198 errors:0 dropped:0 overruns:0 frame:0
          TX packets:624 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:55555 (55.5 KB)  TX bytes:96370 (96.3 KB)

wlp3s0f0  Link encap:Ethernet  HWaddr 48:5a:b6:03:28:7f  
          inet addr:192.168.1.3  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::e6a5:9e76:34f8:fe45/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:657207 errors:0 dropped:0 overruns:0 frame:0
          TX packets:370496 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:883579787 (883.5 MB)  TX bytes:38367961 (38.3 MB)

sudo iptables -L -t nat -n:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:192.168.99.100:32150

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:5000
MASQUERADE  all  --  10.99.0.0/24        !10.99.0.0/24        

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5000 to:172.17.0.2:5000

tailf /var/log/kern.log | grep TRACE: (hangs on the DNAT rule and repeats):

Dec 22 20:14:41 xxx-X550LC kernel: [ 8848.624645] TRACE: raw:OUTPUT:policy:2 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47705 DF PROTO=TCP SPT=51306 DPT=80 SEQ=1122574632 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A00209B710000000001030307) UID=1000 GID=1000 
Dec 22 20:14:41 xxx-X550LC kernel: [ 8848.624659] TRACE: mangle:OUTPUT:policy:1 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47705 DF PROTO=TCP SPT=51306 DPT=80 SEQ=1122574632 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A00209B710000000001030307) UID=1000 GID=1000 
Dec 22 20:14:41 xxx-X550LC kernel: [ 8848.624667] TRACE: nat:OUTPUT:rule:2 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47705 DF PROTO=TCP SPT=51306 DPT=80 SEQ=1122574632 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A00209B710000000001030307) UID=1000 GID=1000 
Dec 22 20:14:42 xxx-X550LC kernel: [ 8849.622139] TRACE: raw:OUTPUT:policy:2 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47706 DF PROTO=TCP SPT=51306 DPT=80 SEQ=1122574632 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A00209C6B0000000001030307) UID=1000 GID=1000 
Dec 22 20:14:42 xxx-X550LC kernel: [ 8849.622169] TRACE: mangle:OUTPUT:policy:1 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47706 DF PROTO=TCP SPT=51306 DPT=80 SEQ=1122574632 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A00209C6B0000000001030307) UID=1000 GID=1000 
Dec 22 20:14:42 xxx-X550LC kernel: [ 8849.622180] TRACE: nat:OUTPUT:rule:2 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47706 DF PROTO=TCP SPT=51306 DPT=80 SEQ=1122574632 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A00209C6B0000000001030307) UID=1000 GID=1000 

Best Answer

The missing pieces to make it work were:

sudo sysctl -w net.ipv4.conf.all.route_localnet=1
sudo iptables -t nat -A POSTROUTING -j MASQUERADE

However, I do not understand why it did work when the destination was 192.168.1.3:32150.

Related Question