Ssh – How to create/setup vpn using only SSH

centosroutesshvpn

Here is the problem I am trying to solve. There is a server ("remote system") that I am able to ssh into from my local computer but this remote system does not have internet connection. I want to provide the remote system with access to the internet through my local computer using ssh based VPN. How do I accomplish this? I have tried the following, which seems to partially work. What I mean by 'partially work' is that connection packets (sync packets) are sent to my local computer but fail to establish the connection to the internet. I am using tcpdump to capture packets on local computer. The local computer and remote system are both running centos 7.

The setup – Note: commands below are run in order. The user@remote commands are run on the remote server and the user@local commands are run on the local computer.

[user@remote ~]$ ip addr show
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether AA:BB:CC:DD:EE:FF brd ff:ff:ff:ff:ff:ff
    inet AAA.BBB.CCC.DDD/24 brd AAA.BBB.CCC.255 scope global dynamic eth0
       valid_lft 1785sec preferred_lft 1785sec
    inet6 EEEE:FFFF:GGGG:HHHH:IIII:JJJJ:KKKK:LLLL/64 scope global noprefixroute dynamic 
       valid_lft 2591987sec preferred_lft 604787sec
    inet6 ABCD::IIII:JJJJ:KKKK:LLLL/64 scope link 
       valid_lft forever preferred_lft forever
[user@local ~]$ ip addr show
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether AA:BB:CC:DD:EE:FF brd ff:ff:ff:ff:ff:ff
    inet AAA.BBB.CCC.DDD/24 brd AAA.BBB.CCC.255 scope global dynamic eth0
       valid_lft 1785sec preferred_lft 1785sec
    inet6 EEEE:FFFF:GGGG:HHHH:IIII:JJJJ:KKKK:LLLL/64 scope global noprefixroute dynamic 
       valid_lft 2591987sec preferred_lft 604787sec
    inet6 ABCD::IIII:JJJJ:KKKK:LLLL/64 scope link 
       valid_lft forever preferred_lft forever

Create the tun0 interface on the remote system.

[user@remote ~]$ sudo ip tuntap add tun0 mode tun
[user@remote ~]$ ip addr show
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether AA:BB:CC:DD:EE:FF brd ff:ff:ff:ff:ff:ff
    inet AAA.BBB.CCC.DDD/24 brd AAA.BBB.CCC.255 scope global dynamic eth0
       valid_lft 1785sec preferred_lft 1785sec
    inet6 EEEE:FFFF:GGGG:HHHH:IIII:JJJJ:KKKK:LLLL/64 scope global noprefixroute dynamic 
       valid_lft 2591987sec preferred_lft 604787sec
    inet6 ABCD::IIII:JJJJ:KKKK:LLLL/64 scope link 
       valid_lft forever preferred_lft forever
3: tun0:  mtu 1500 qdisc noop state DOWN qlen 500
    link/none 

Create the tun0 interface on the local system.

[user@local ~]$ sudo ip tuntap add tun0 mode tun
[user@local ~]$ ip addr show
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether AA:BB:CC:DD:EE:FF brd ff:ff:ff:ff:ff:ff
    inet AAA.BBB.CCC.DDD/24 brd AAA.BBB.CCC.255 scope global dynamic eth0
       valid_lft 1785sec preferred_lft 1785sec
    inet6 EEEE:FFFF:GGGG:HHHH:IIII:JJJJ:KKKK:LLLL/64 scope global noprefixroute dynamic 
       valid_lft 2591987sec preferred_lft 604787sec
    inet6 ABCD::IIII:JJJJ:KKKK:LLLL/64 scope link 
       valid_lft forever preferred_lft forever
3: tun0:  mtu 1500 qdisc noop state DOWN qlen 500
    link/none

Assign an ip address to tun0 and bring it up:

[user@local ~]$ sudo ip addr add 10.0.2.1/30 dev tun0
[user@local ~]$ sudo ip link set dev tun0 up
[user@local ~]$ ip addr show tun0
3: tun0:  mtu 1500 qdisc pfifo_fast state DOWN qlen 500
    link/none 
    inet 10.0.2.1/30 scope global tun0
       valid_lft forever preferred_lft forever
[user@remote ~]$ sudo ip addr add 10.0.2.2/30 dev tun0
[user@remote ~]$ sudo ip link set dev tun0 up
[user@remote ~]$ ip addr show tun0
3: tun0:  mtu 1500 qdisc pfifo_fast state DOWN qlen 500
    link/none 
    inet 10.0.2.2/30 scope global tun0
       valid_lft forever preferred_lft forever

Modify sshd_config on both the remote and local systems to enable tunneling:

[user@remote ~]$ sudo grep PermitTunnel /etc/ssh/sshd_config 
PermitTunnel point-to-point
[user@local ~]$ sudo grep PermitTunnel /etc/ssh/sshd_config 
PermitTunnel point-to-point

Create the ssh tunnel:

[user@local ~]$ sudo ssh -f -w0:0 root@remote true
root@remote's password: 
[user@local ~]$ ps aux | grep root@remote
root      1851  0.0  0.0  76112  1348 ?        Ss   23:12   0:00 ssh -f -w0:0 root@remote true

Test ping on both server using the new ip addresses:

[user@local ~]$ ping 10.0.2.2 -c 2
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.
64 bytes from 10.0.2.2: icmp_seq=1 ttl=64 time=1.68 ms
64 bytes from 10.0.2.2: icmp_seq=2 ttl=64 time=0.861 ms

--- 10.0.2.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.861/1.274/1.688/0.415 ms
[user@remote ~]$ ping 10.0.2.1 -c 2
PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
64 bytes from 10.0.2.1: icmp_seq=1 ttl=64 time=0.589 ms
64 bytes from 10.0.2.1: icmp_seq=2 ttl=64 time=0.889 ms

--- 10.0.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.589/0.739/0.889/0.150 ms
[user@remote ~]$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    100    0        0 eth0
10.0.2.0        0.0.0.0         255.255.255.252 U     0      0        0 tun0
AAA.BBB.CCC.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
[user@remote ~]$ ip route show
default via AAA.BBB.CCC.1 dev eth0  proto static  metric 100 
10.0.2.0/30 dev tun0  proto kernel  scope link  src 10.0.2.2 
AAA.BBB.CCC.0/24 dev eth0  proto kernel  scope link  src AAA.BBB.CCC.31  metric 100 

Get google ip addresses:

[user@local ~]$ nslookup google.com
Server:     s.e.r.ver
Address:    s.e.r.ver#53

Non-authoritative answer:
Name:   google.com
Address: 173.194.219.101
Name:   google.com
Address: 173.194.219.100
Name:   google.com
Address: 173.194.219.113
Name:   google.com
Address: 173.194.219.102
Name:   google.com
Address: 173.194.219.139
Name:   google.com
Address: 173.194.219.138

IMPORTANT: I ran the above command at a different time and got a different result. Do not assume that your response will be same as mine for nslookup above.

Since all of google's ip addresses begin with 173.194.219, lets route all these ip addresses through the local computer.

[user@remote ~]$ sudo ip route add 173.194.219.0/24 dev tun0
[user@remote ~]$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    100    0        0 eth0
10.0.2.0        0.0.0.0         255.255.255.252 U     0      0        0 tun0
AAA.BBB.CCC.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
173.194.219.0   0.0.0.0         255.255.255.0   U     0      0        0 tun0
[user@remote ~]$ ip route show
default via AAA.BBB.CCC.1 dev eth0  proto static  metric 100 
10.0.2.0/30 dev tun0  proto kernel  scope link  src 10.0.2.2 
AAA.BBB.CCC.0/24 dev eth0  proto kernel  scope link  src AAA.BBB.CCC.31  metric 100 
173.194.219.0/24 dev tun0  scope link 

Enable ip_forwarding:

[user@local ~]$ grep ip_forward /etc/sysctl.conf 
net.ipv4.ip_forward = 1
[user@local ~]$ sudo service network restart
Restarting network (via systemctl):                        [  OK  ]

Setup packet capture on local computer using tcpdump:

[user@local ~]$ sudo tcpdump -nn -vv 'port not 22' -i any
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

Attempt to connect to google from remote server.

[user@remote ~]$ openssl s_client -connect google.com:443
socket: No route to host
connect:errno=113

As soon as the openssl command is run on the remote server, the tcpdump captures some packets:

    10.0.2.2.52768 > 173.194.219.102.443: Flags [S], cksum 0x8702 (correct), seq 994650730, win 29200, options [mss 1460,sackOK,TS val 7701438 ecr 0,nop,wscale 7], length 0
00:49:33.247753 IP (tos 0x0, ttl 64, id 46037, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.48774 > 173.194.219.100.443: Flags [S], cksum 0x47a7 (correct), seq 2218733674, win 29200, options [mss 1460,sackOK,TS val 7701439 ecr 0,nop,wscale 7], length 0
00:49:33.247883 IP (tos 0xc0, ttl 64, id 9538, offset 0, flags [none], proto ICMP (1), length 88)
    10.0.2.1 > 10.0.2.2: ICMP host 173.194.219.100 unreachable - admin prohibited, length 68
    IP (tos 0x0, ttl 63, id 46037, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.48774 > 173.194.219.100.443: Flags [S], cksum 0x47a7 (correct), seq 2218733674, win 29200, options [mss 1460,sackOK,TS val 7701439 ecr 0,nop,wscale 7], length 0
00:49:33.253068 IP (tos 0x0, ttl 64, id 26282, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.51312 > 173.194.219.101.443: Flags [S], cksum 0x6ff8 (correct), seq 2634016105, win 29200, options [mss 1460,sackOK,TS val 7701443 ecr 0,nop,wscale 7], length 0
00:49:33.254771 IP (tos 0xc0, ttl 64, id 9539, offset 0, flags [none], proto ICMP (1), length 88)
    10.0.2.1 > 10.0.2.2: ICMP host 173.194.219.101 unreachable - admin prohibited, length 68
    IP (tos 0x0, ttl 63, id 26282, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.51312 > 173.194.219.101.443: Flags [S], cksum 0x6ff8 (correct), seq 2634016105, win 29200, options [mss 1460,sackOK,TS val 7701443 ecr 0,nop,wscale 7], length 0
00:49:33.258805 IP (tos 0x0, ttl 64, id 9293, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.33686 > 173.194.219.139.443: Flags [S], cksum 0x542b (correct), seq 995927943, win 29200, options [mss 1460,sackOK,TS val 7701450 ecr 0,nop,wscale 7], length 0
00:49:33.258845 IP (tos 0xc0, ttl 64, id 9540, offset 0, flags [none], proto ICMP (1), length 88)
    10.0.2.1 > 10.0.2.2: ICMP host 173.194.219.139 unreachable - admin prohibited, length 68
    IP (tos 0x0, ttl 63, id 9293, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.33686 > 173.194.219.139.443: Flags [S], cksum 0x542b (correct), seq 995927943, win 29200, options [mss 1460,sackOK,TS val 7701450 ecr 0,nop,wscale 7], length 0
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel

The packets captured using tcpdump suggest that an attempt is made to establish the connection (Sync packets are sent) but nothing is received. Also there is a message 10.0.2.1 > 10.0.2.2: ICMP host 173.194.219.139 unreachable - admin prohibited, length 68 that seems to suggest a problem.

Any suggestions on how to work around this issue? Are there iptable rules that need to be added? Any firewall issues (firewall-d?).


Note #1
Output from iptables-save:

[user@local ~]$ sudo iptables -t nat -A POSTROUTING -s 10.0.2.2/32 ! -d 10.0.2.1/30 -j MASQUERADE -o eth0
[user@local ~]$ sudo iptables-save
# Generated by iptables-save v1.4.21 on Sat Apr 15 01:40:57 2017
*nat
:PREROUTING ACCEPT [35:8926]
:INPUT ACCEPT [1:84]
:OUTPUT ACCEPT [6:439]
:POSTROUTING ACCEPT [6:439]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING -s 10.0.2.2/32 ! -d 10.0.2.0/30 -j MASQUERADE
-A POSTROUTING_ZONES -o eth0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i eth0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Sat Apr 15 01:40:57 2017
# Generated by iptables-save v1.4.21 on Sat Apr 15 01:40:57 2017
*mangle
:PREROUTING ACCEPT [169:18687]
:INPUT ACCEPT [144:11583]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [80:8149]
:POSTROUTING ACCEPT [80:8149]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i eth0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Sat Apr 15 01:40:57 2017
# Generated by iptables-save v1.4.21 on Sat Apr 15 01:40:57 2017
*security
:INPUT ACCEPT [2197:163931]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1229:185742]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Sat Apr 15 01:40:57 2017
# Generated by iptables-save v1.4.21 on Sat Apr 15 01:40:57 2017
*raw
:PREROUTING ACCEPT [2362:184437]
:OUTPUT ACCEPT [1229:185742]
:OUTPUT_direct - [0:0]
:PREROUTING_direct - [0:0]
-A PREROUTING -j PREROUTING_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Sat Apr 15 01:40:57 2017
# Generated by iptables-save v1.4.21 on Sat Apr 15 01:40:57 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [80:8149]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i eth0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o eth0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i eth0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Sat Apr 15 01:40:57 2017


Note #2:
I setup an apache webserver on a separate host that only the local server has access to. I ran tcpdump on webserver listening on port 80. When I run telnet webserver 80 I capture the following packets. This is expected behavior since TCP Connection is established (S, S-Ack, Ack).

[user@webserver ~]$ sudo tcpdump -nn -vv 'port not 22 and 80' -i eth0 
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
07:17:30.411474 IP (tos 0x10, ttl 64, id 34376, offset 0, flags [DF], proto TCP (6), length 60)
    local.server.46710 > web.server.80: Flags [S], cksum 0x8412 (incorrect -> 0x6d96), seq 3018586542, win 29200, options [mss 1460,sackOK,TS val 3047398 ecr 0,nop,wscale 7], length 0
07:17:30.411557 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    web.server.80 > local.server.46710: Flags [S.], cksum 0x8412 (incorrect -> 0x9114), seq 2651711943, ack 3018586543, win 28960, options [mss 1460,sackOK,TS val 37704524 ecr 3047398,nop,wscale 7], length 0
07:17:30.411725 IP (tos 0x10, ttl 64, id 34377, offset 0, flags [DF], proto TCP (6), length 52)
    local.server.46710 > web.server.80: Flags [.], cksum 0x840a (incorrect -> 0x301c), seq 1, ack 1, win 229, options [nop,nop,TS val 3047398 ecr 37704524], length 0

When I attempt to connect to the webserver from the remote server through the local server, tcpdump on webserver does not capture any packets (not even the initial Sync) but local server captures Sync packet being sent to webserver (see below). This makes me believe that something is preventing the packets from being sent to webserver – maybe a misconfiguration or firewall.

[user@local ~]$ sudo tcpdump -nn -vv 'port not 22 and 80' -i any
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
02:24:09.135842 IP (tos 0x10, ttl 64, id 38062, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.50558 > web.server.80: Flags [S], cksum 0x668d (correct), seq 69756226, win 29200, options [mss 1460,sackOK,TS val 4780524 ecr 0,nop,wscale 7], length 0

IMPORTANT: the packets are not being routed through eth0 but instead attempt is made to send the packets to the webserver via tun0 (which fails). I can confirm this by running tcpdump on tun0 interface:

[user@local ~]$ sudo tcpdump -nn -vv 'port not 22 and 80' -i tun0
tcpdump: listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
02:28:10.295972 IP (tos 0x10, ttl 64, id 46976, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.50560 > webserver.80: Flags [S], cksum 0xd560 (correct), seq 605366388, win 29200, options [mss 1460,sackOK,TS val 5021684 ecr 0,nop,wscale 7], length 0


Note #3:
I turned off firewalld in the local computer and Sync packets were received by the webserver.

[user@local ~]$ sudo systemctl stop firewalld
[user@webserver ~]$ sudo tcpdump -nn -vv 'port not 22 and 80' -i eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:25:17.390912 IP (tos 0x10, ttl 63, id 61767, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.50580 > web.server.80: Flags [S], cksum 0x30dc (correct), seq 2601927549, win 29200, options [mss 1460,sackOK,TS val 7123514 ecr 0,nop,wscale 7], length 0
08:25:17.391003 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    web.server.80 > 10.0.2.2.50580: Flags [S.], cksum 0x4e23 (incorrect -> 0xa316), seq 959115533, ack 2601927550, win 28960, options [mss 1460,sackOK,TS val 41771503 ecr 7123514,nop,wscale 7], length 0
08:25:17.391192 IP (tos 0x0, ttl 128, id 60032, offset 0, flags [none], proto TCP (6), length 40)
    10.0.2.2.50580 > web.server.80: Flags [R], cksum 0x7339 (correct), seq 2601927550, win 8192, length 0
08:25:18.393794 IP (tos 0x10, ttl 63, id 61768, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.2.2.50580 > web.server.80: Flags [S], cksum 0x2cf1 (correct), seq 2601927549, win 29200, options [mss 1460,sackOK,TS val 7124517 ecr 0,nop,wscale 7], length 0
08:25:18.393898 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    web.server.80 > 10.0.2.2.50580: Flags [S.], cksum 0x4e23 (incorrect -> 0x7e71), seq 974785773, ack 2601927550, win 28960, options [mss 1460,sackOK,TS val 41772506 ecr 7124517,nop,wscale 7], length 0
08:25:18.394003 IP (tos 0x0, ttl 128, id 60033, offset 0, flags [none], proto TCP (6), length 40)
    10.0.2.2.50580 > web.server.80: Flags [R], cksum 0x566a (correct), seq 2601927550, win 8192, length 0

Now clearly, the source IP needs to be updated to match the local server's ip address before the packet is sent to the webserver. As @xin suggested, NAT needs to be setup on the local server.


Note #4:
Once I attempt to connect to the webserver, I can see that the pkts count for rule 9 goes up by 1 (as seen below).

[user@local ~]$ sudo iptables -nvL --line-numbers
..........
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
2        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
3        1    60 FORWARD_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
4        1    60 FORWARD_IN_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
5        1    60 FORWARD_IN_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
6        1    60 FORWARD_OUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
7        1    60 FORWARD_OUT_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
8        0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
9        1    60 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
..........
[user@local ~]$ sudo iptables -D FORWARD 9

Once rule 9 from the FORWARD chain is deleted (from above, as @xin suggested), I am able to connect to the webserver.

[user@local ~]$ sudo iptables -nvL --line-numbers
..........
Chain FORWARD (policy ACCEPT 1 packets, 60 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1       12  5857 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
2        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
3        2   120 FORWARD_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
4        2   120 FORWARD_IN_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
5        2   120 FORWARD_IN_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
6        2   120 FORWARD_OUT_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
7        2   120 FORWARD_OUT_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
8        0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
..........

Best Answer

Source address of packets should be replaced by one of local machine's address so that answers could be received by local machine, otherwise there is no (good) reason for sending these packets to next routers, answer could not be caught anyway. iptables MASQUERADE and SNAT are useful to change source address of these packets:

[user@local ~]$ iptables -t nat -A POSTROUTING -s 10.0.2.2/32 ! -d 10.0.2.1/30 -j MASQUERADE -o eth0
Related Question