Ubuntu – 16.10 fail to resolve DNS

16.10dnsnetworkingresolv.confsystemd

After upgrading my 16.04-installation to 16.10, I have trouble with DNS.

First I got problems a couple of times when connected to WiFi, while it worked on ethernet. Now it seems to work on WiFi also. Not sure why, and if it is in any way related to the problem I face now:

When connecting to a VPN host with Cisco Anyconnect VPN, it adds a line in '/etc/resolv.conf'. I understand that Ubuntu is now using systemd-resolve, and the man page says that there are three different modes for handling /etc/resolv.conf. My /etc/resolv.conf is not a symlink, and does not list 127.0.0.53 as a DNS server, so as far as I understand systemd-resolved should "read it for DNS configuration data". However, it does not seem to care about it.

dig

The strange thing (for me) is that dig host.customer.tld, returns a nice answer with an ANSWER SECTION showing the ip of the requested host, and it refers to the dns server added to /etc/resolv.conf by vpn client as the SERVER. When vpn connection is disabled I get no answer. I.e. dig reads /etc/resolv.conf.

ping

The browser, on the other side, does not get to /etc/resolv.conf, and is not able to resolve the host name. Neither is ping/curl, by the way.

nmcli

I found a related post, and tried running

nmcli device show <interfacename> | grep IP4.DNS

but it lists no dns for the cscotun0 device. (It does not in 16.04 neither, though.) Also, nmcli lists my dhcp server (my router) as IP4.DNS host for my eth/wlan connections. Using dig @192.168.0.1 xxx for any public domain works fine.

configuration

There are some other DNS servers listed in my /run/systemd/resolve/resolv.conf:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
# Too many DNS servers configured, the following entries may be ignored.
nameserver 2001:4860:4860::8844

These are not served by my DHCP server. the file /etc/systemd/resolved.conf contains only commented lines, except the section header:

[Resolve]
#DNS=
#FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844

The man page for resolved.conf says that

DNS=
A space-separated list of IPv4 and IPv6 addresses to use as system DNS servers.
…
For compatibility reasons, if this setting is not specified, the DNS servers
listed in /etc/resolv.conf are used instead, if that file exists and any
servers are configured in it. This setting defaults to the empty list.

FallbackDNS=
A space-separated list of IPv4 and IPv6 addresses to use as the fallback DNS
servers. Any per-link DNS servers obtained from systemd-networkd.service(8)
take precedence over this setting, as do any servers set via DNS= above or
/etc/resolv.conf. This setting is hence only used if no other DNS server
information is known. If this option is not given, a compiled-in list of DNS servers is used instead.

Seems like the fallback ends up in /run/systemd/resolve/resolv.conf in my case.

EDIT: I was not certain what was the problem, and to be honest I still don't know exactly how this works, but at least it turned out that the solution in my case was to disable the systemd-resolved service. I thought that service was required, that it was the component that provided DNS service to all the local applications, but apparently there are something else in there doing that job.

Best Answer

I experienced similar problems, for example with adding an extra USB wifi dongle. First I disabled dnsmasq in networkmanager as described above and I stopped dnsmasq (service dnsmasq stop)

I noticed that when resolving broke during my VPN connecting, the routing table looks slightly different (output of route command). The name of the Gateway is DD-WRT in the case it does not work and simply 'gateway' when it does work. The output of this did not change:

nmcli device show wlp1s0 | grep IP4.DNS

It kept showing my router IP. A workaround to get it to work for a while is to restart systemd-resolvd:

sudo service systemd-resolved restart

Since dnsmasq is out of the equation, it is either systemd-resolvd that is the cause of the issue, or anything changing the routing table.

So this is the only difference I see:

ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    601    0        0

which works. And this when it does NOT work:

ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         DD-WRT          0.0.0.0         UG    601    0        0 wlp1s0

And the same name difference on the VPN line :

vpn-dns.name gateway         255.255.255.255 UGH   0      0        0 wlp1s0

Who knows what may influence the routing table? It would be great if we can identify this so a bug report can be filed. I am getting seriously sick and tired of pursuing all these bugs, but I would like to get them fixed so future users and us will be happy :).

[update] It seems stopping systemd-resolved may fix this and not negatively impact other stuff. You can try that and let it know if it does break stuff. I saw when running systemd-resolvd in debug when it broke:

Removing scope on link wlp1s0, protocol llmnr, family AF_INET
Removing scope on link wlp1s0, protocol llmnr, family AF_INET6
Removing scope on link *, protocol dns, family *

To disable:

sudo systemctl disable systemd-resolved.service

I updated the Ubuntu report with suggestions. [/update] Add: Note: the bug report : https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317 has a patch for 17.04 for some issues. Please check the bug report and if possible test the patch. Thank you!

[update]

Please check the above mentioned bug report, the issue seems to be resolved for 17.10 and with a simple command DNS leakage can be disabled too.

[/update]

DNSoverTLS 1.1.1.1 OpenVPN configuration.

First you need systemd-resolved installed and configured to use stub-resolv.conf.

ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
cat /etc/resolv.conf

Output

nameserver 127.0.0.53
options edns0

systemd-networkd

/etc/systemd/resolved.conf (example):

[Resolve]
DNS=8.8.8.8 8.8.4.4
FallbackDNS=1.1.1.1 1.0.0.1
LLMNR=no
MulticastDNS=no
DNSSEC=allow-downgrade
DNSOverTLS=opportunistic
Cache=yes
DNSStubListener=yes

/etc/systemd/network/ethX.network (example):

[Match]
Name=eth*

[Link]
RequiredForOnline=yes

[Network]
DHCP=yes
MulticastDNS=no
LLMNR=no
LinkLocalAddressing=no

[DHCP]
UseDNS=yes
UseHostname=no
CriticalConnection=yes

/etc/systemd/network/tunX.network (important!):

(in order for openvpn to be able to administer tun link, the link must be unmanaged)

[Match]
Name=tun*

[Link]
Unmanaged=yes

I use update-resolved to configure systemd-resolved. (you can use update-systemd-resolved or aptitude install openvpn-systemd-resolved, but when you need to follow README.md instead).

Installing update-resolved:

cd /etc/openvpn
git clone https://github.com/bac0n/update-resolved.git

Add update-resolved to your openvpn.conf:

# Include update-resolved up/down script.
config /etc/openvpn/update-resolved/update-resolved.ovpn

Restart openvpn:

systemctl restart openvpn

Journald:

journalctl -t update-resolved

Output

-- Logs begin at Sat 2019-09-21 12:28:01 CEST, end at Sun 2019-09-22 17:05:01 CEST. --
Sep 21 12:28:11 foobar update-resolved[914]: Note: Successfully configured resolved on link 3 (tun0)

Note:

As default it uses openvpn supplied dns´s. if you like to use static dns´s you need to filter the dns´s supplied by openvpn in 'update-resolved.ovpn' and set your own dns´s in 'update-resolved.conf'

Example:

resolve_options=(DOMAIN ~. DNS 1.1.1.1 DNS 1.0.0.1 LLMNR no MulticastDNS no)

(when using domain ~. resolved will use the tun link for all your dns queries (unless other too carry such a route-only domain). When the tun link is removed resolved will start using 'global' and 'isp' dns´s in parallel, Protocols and Routing)

Ubuntu – Openconnect in network-manager does not update resolv.conf

I found the solution. Basically, if the dns server sents ipv6 dns servers to lookup ipv4 addresses things go wrong. Openconnect will put ipv6 addresses in INTERNAL_IP4_DNS and the "network-manager-openconnect" does not expect that, treats the whole variable (and basically all dns servers) as garbage and goes on. I compiled my own network-manager-openconnect from master which has a fix for this, and that works fine.

I don't know why I have this problem after upgrading. Maybe something changed in openconnect? Or maybe in the day I upgraded my company network admin added an ipv6 dns server? (I think that is unlikely...)

If you are using network-manager-openconnect 1.2.6 or 1.2.7-dev (or maybe even a lower version, which does not contain the fix) you can compile your own version from master like so:

sudo apt-get build-dep network-manager-openconnect
mkdir ~/network-manager-openconnect_build
cd ~/network-manager-openconnect_build
git clone https://gitlab.gnome.org/GNOME/NetworkManager-openconnect.git
cd NetworkManager-openconnect
./autogen.sh
make
sudo mv /usr/lib/NetworkManager/nm-openconnect-service-openconnect-helper /usr/lib/NetworkManager/nm-openconnect-service-openconnect-helper_bak
sudo cp src/nm-openconnect-service-openconnect-helper /usr/lib/NetworkManager/nm-openconnect-service-openconnect-helper

Note: I only copied over the helper bin. In theory this could give incompatibility problems. I had no problems with it. But if you have, you might try to copy over the main bin as wel from src.

If you want, you can use the following to have extra logging for the vpn module to see in the syslog if openconnect received any dns data:

sudo nmcli general logging level KEEP domains VPN_PLUGIN:TRACE

If you see the INTERNAL_IP4_DNS being set with both ipv4 and ipv6 addresses, and you use the openconnect network manager version mentioned above, you are affected with this bug.

dig