I have a Kubernetes cluster running applications (currently on a set of Vagrant CoreOS VMs on a local server)
I want to be able to debug a particular application locally on my laptop, so I worked on setting up VPN into the cluster: a client/server VPN based on kylemanna/docker-openvpn
, deployed as a regular Pod
I created the cert/key pairs, client certs etc…
I can connect to the VPN fine.
Now, connecting to the VPN server doesn't get me much if I can't access the Services.
I have the DNS addon running skyDNS in the cluster. I can nslookup
my services from other pods in the cluster, so all that works fine, but I can't resolve Services by name on the VPN client.
I can ping Pods by IP from the VPN client (in the subnet 10.2.0.0/16
) but I can't resolve with DNS
a nslookup
from the client returns:
nslookup myservice 10.3.0.10
Server: 10.3.0.10
Address: 10.3.0.10#53
** server can't find myservice: SERVFAIL
One of the problems of troubleshooting is that neither ping
nor traceroute
work on the DNS IP (from any pod), yet it resolves services, so nslookup
is the way I know to check, but that is not very informative.
The VPN host IP the Pod binds to is 192.168.10.152
The Kubernetes subnet is 10.2.0.0/16
The SkyDNS server is at 10.3.0.10
The VPN server subnet is 10.8.0.0/24
On the VPN server ifconfig gives:
eth0 Link encap:Ethernet HWaddr 02:42:0A:02:16:45
inet addr:10.2.22.69 Bcast:0.0.0.0 Mask:255.255.255.0
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
So 10.2.22.69
is the Pod IP and the VPN Server IP is 10.8.0.1
with the Gateway being 10.8.0.2
i guess.
On the VPN server pod the routign table looks like:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.2.22.1 0.0.0.0 UG 0 0 0 eth0
10.2.22.0 * 255.255.255.0 U 0 0 0 eth0
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
10.8.0.2 * 255.255.255.255 UH 0 0 0 tun0
192.168.254.0 10.8.0.2 255.255.255.0 UG 0 0 0 tun0
I can reach my applications by IP (and get data) but couldn't when using the service IP (the proxy IP which is on the 10.3.0.0 subnet)
I added the route route add -net 10.3.0.0/16 gw 10.8.0.2
to the VPN Server and I can then use the Service IP to get data but the nslookup
just times out then.
I guess the traffic may not be coming back from the DNS. DNS is itself a proxied service in Kubernetes, so that adds a level of complexity.
Not sure how to fix this.
Best Answer
finally my config looks like this:
-u
for the VPN server address and port-n
for all the DNS servers to use-s
to define the VPN subnet (as it defaults to 10.2.0.0 which is used by Kubernetes already)-d
to disable NAT-p
to push options to the client-N
to enable NAT: it seems critical for this setup on Kubernetesthe last part, pushing the search domains to the client, was the key to getting
nslookup
etc.. to work.note that curl didn't work at first, but seems to start working after a few seconds. So it does work but it takes a bit of time for curl to be able to resolve.