MacOS – OS X doesn’t query local DNS names properly

dnsmacosNetworkvpn

None of our OS X client machines in our workplace query local DNS names of our servers properly after connecting to VPN. For other client machines – including Linux and Windows – it works properly.

Just some context: our servers were accessible publicly before. Since we started securing everything, we proceed to put them behind a firewall and they are accessible via VPN only. Since our developers are used to accessing the servers via fqdn, we added a local DNS server. So when they connect to our VPN, DHCP + DNS settings would be passed to their machines automatically. Two DNS server IPs, one is for local resolving and one for public resolving. Unfortunately, all of our OS X users are experiencing these problems. Linux and Windows machines aren't.

Server: riker.example.com

  • 120.x.x.x (Public IP address)
  • 10.20.1.28 (Local IP address)

DNS settings that is passed to our users:

  • 10.20.1.3 (for local)
  • 8.8.8.8 (for public)

We have already checked:

  • DNS Server settings of our Fortigate device – we're 100% sure it is correct because our Linux and Windows clients are working.
  • Firewall policies – again, 100% sure because it's working properly with Linux and Windows
  • Check /etc/resolv.conf files of the OS X machines – correct DNS settings given by DNS server [edit June 6, 2016]
  • Flushed DNS cache using this: sudo killall -HUP mDNSResponder
  • Turned off firewall of OS X machines
  • Restarted mDNSresponder

After all these troubleshooting steps, it still does not work.

If I'm connected to VPN, it's still resolving the public IP address, and not the local IP address :

heineken:~ heineken$ ping riker.example.com
PING riker.example.com (120.x.x.x): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- riker.example.com ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
heineken:~ heineken$

But if I ping the local address :

heineken:~ heineken$ ping 10.20.1.28
PING 10.20.1.28 (10.20.1.28): 56 data bytes
64 bytes from 10.20.1.28: icmp_seq=0 ttl=63 time=62.714 ms
64 bytes from 10.20.1.28: icmp_seq=1 ttl=63 time=83.162 ms
^C
--- 10.20.1.28 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 62.714/72.938/83.162/10.224 ms

Checking by dig and nslookup (still connected to VPN):

heineken:~ heineken$ dig @10.20.1.3 riker.example.com
 
; <<>> DiG 9.8.3-P1 <<>> @10.20.1.3 riker.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64601
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
 
;; QUESTION SECTION:
;riker.example.come. IN A
 
;; ANSWER SECTION:
riker.example.com. 3600 IN A 10.20.1.28
 
;; Query time: 58 msec
;; SERVER: 10.20.1.3#53(10.20.1.3)
;; WHEN: Fri Jun 10 11:33:00 2016
;; MSG SIZE  rcvd: 52
 
heineken:~ heineken$ nslookup riker.example.com
Server: 10.20.1.3
Address: 10.20.1.3#53
 
Non-authoritative answer:
Name: riker.example.com
Address: 10.20.1.28

Contents of /etc/resolv.conf of local machine (still connected to VPN):

heineken:~ heineken$ cat /etc/resolv.conf
#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
nameserver 10.20.1.3
nameserver 8.8.8.8

As you can see above, by pinging the domain it's still gives out the public IP address and not the local IP address.
But with dig and nslookup, it's already giving out the local IP Address.

Few more things I did:

  • Tried the solutions presented from here:
    DNS not resolving on Mac OS X – still did not work for us.
  • I installed a virtual machine CentOS – inside an OS X machine and made it to have a bridge connection. It was resolving correctly the domain locally.
    • This means that Forticlient can be eliminated from the equation since the OS X and CentOS VM is just sharing the same adapter.
  • Seeked L3 support from Fortigate
    • We did a lot of tcpdump on the Fortigate device to check traffic.
      • We discovered that when a Linux or a Windows machine pings a domain it seeks for DNS request from the Fortigate device first.
      • Unfortunately for the OS X machine, it does not do any DNS request from the Fortigate device. So what happens is that it still resolves to the public domain.
    • I was advised by them that the issue is localized on the OS X devices because even after flushing cache, it does not perform any DNS query when pinging riker.example.com. So they cannot help us anymore.

This was tested on:
OS X versions: Yosemite 10.10.4, El Capitan 10.11.5
Fortigate 100D Firmware version: v5.2.4,build688 (GA)
Forticlient version: 5.4.0.493

Any advice?

Best Answer

I think the key to your issue might be that your internal DNS is "non-authorative" (see the output from nslookup), suggested in the answer to this question: DNS resolves servers external IP instead of internal IP

If you run nslookup -type=soa riker.example.com is the "origin" server your local or public DNS server?

Related Question