Ubuntu – need to add nameservers to resolv.conf

dnssystemd-resolved

I'm running Ubuntu 18.04 (upgraded from some earlier version) which uses Network Manager and systemd-resolved for name resolution. When I boot, my ethernet connection enp0s31f6 is brought up by Network Manager and given three nameserver addresses via DHCP, 10.1.13.10, 10.1.141.10, 10.1.13.36. Running nmcli shows the three nameservers under "DNS configuration". Running systemd-resolve --status shows them under a "Link 2 (enp0s31f6)" section. I can ping each one. No other connection is active.

testuser ☼ systemd-resolve --status
Global
          DNS Domain: (my org's domain)
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (wlp4s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (enp0s31f6)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.1.13.10
                      10.1.141.10
                      10.1.13.36
          DNS Domain: (my org's domain)

However, when I actually try to resolve a name, even the name of one of the nameservers, dig claims that "connection timed out: no servers could be reached".

testuser ☼ dig dcpdc001.(my org's domain)

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> dcpdc001.(my org's domain)
;; global options: +cmd
;; connection timed out; no servers could be reached

Note that this name should resolve to 10.1.13.10, the first nameserver.

I have configured resolvconf to use dynamic updates. /etc/resolv.conf points to /run/resolvconf/resolv.conf. This file contains only (non-comments):

nameserver 127.0.0.53
search (my orgs local search domain)

If I add nameserver 10.1.13.10 to this file manually, suddenly dig can resolve again, and anything else that needs to see local names can do so. Removing the nameserver breaks that again.

I don't know much about the servers. They're part of a Windows-based network, but I can use them if I edit resolv.conf manually so I don't think that's the issue, and it implies I don't need to be authenticated to the domain to use them. (I can authenticate to the domain via Ubuntu using Realmd/SSSD, but not if I can't resolve the domain controller…)

The journalctl entries for systemd-resolved show only a few messages about "Using degraded feature set … for DNS server" but they only refer to the third nameserver, not the others. Nothing for the primary nameserver.

How can I get name resolution working without having to manually edit resolv.conf every time I boot?

I assume the contents of my resolv.conf mean that Network Manager or Systemd has some sort of local caching resolver running? If so, would bypassing it fix things?

I increased the logging level of systemd-resolved and journalctl -f -u systemd-resolved shows:

Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Got DNS stub UDP query packet for id 19836
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Looking up RR for dcpdc001.(org domain) IN A.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Switching to DNS server 10.1.13.10 for interface enp0s31f6.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Cache miss for dcpdc001.(org domain) IN A
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Transaction 12728 for <dcpdc001.(org domain) IN A> scope dns on enp0s31f6/*.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Using feature level UDP+EDNS0+DO+LARGE for transaction 12728.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Using DNS server 10.1.13.10 for transaction 12728.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Sending query packet with id 12728.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Processing query...
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Timeout reached on transaction 12728.

Best Answer

Systemd comes with a "stub" resolver, systemd-resolved, which according to them is not actually meant to be used as a DNS server:

Well, resolved is not supposed to be a DNS server, it's supposed to be exactly good enough so that libc-like DNS clients can resolve their stuff, and we carry enough info for the AD bit to be set.

For whatever reason, Ubuntu is configured to use it as a DNS server and, in fact, the only one.

A comment on bug #1624320 points out that systemd-resolved has three modes of operation, and the second one is what fixed my problem. Namely:

$ sudo rm -f /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Related Solutions

DNS – 16.10 Fails to Resolve

I experienced similar problems, for example with adding an extra USB wifi dongle. First I disabled dnsmasq in networkmanager as described above and I stopped dnsmasq (service dnsmasq stop)

I noticed that when resolving broke during my VPN connecting, the routing table looks slightly different (output of route command). The name of the Gateway is DD-WRT in the case it does not work and simply 'gateway' when it does work. The output of this did not change:

nmcli device show wlp1s0 | grep IP4.DNS

It kept showing my router IP. A workaround to get it to work for a while is to restart systemd-resolvd:

sudo service systemd-resolved restart

Since dnsmasq is out of the equation, it is either systemd-resolvd that is the cause of the issue, or anything changing the routing table.

So this is the only difference I see:

ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    601    0        0

which works. And this when it does NOT work:

ubuntu@ubuntu-Lenovo-Yoga-2-11:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         DD-WRT          0.0.0.0         UG    601    0        0 wlp1s0

And the same name difference on the VPN line :

vpn-dns.name gateway         255.255.255.255 UGH   0      0        0 wlp1s0

Who knows what may influence the routing table? It would be great if we can identify this so a bug report can be filed. I am getting seriously sick and tired of pursuing all these bugs, but I would like to get them fixed so future users and us will be happy :).

[update] It seems stopping systemd-resolved may fix this and not negatively impact other stuff. You can try that and let it know if it does break stuff. I saw when running systemd-resolvd in debug when it broke:

Removing scope on link wlp1s0, protocol llmnr, family AF_INET
Removing scope on link wlp1s0, protocol llmnr, family AF_INET6
Removing scope on link *, protocol dns, family *

To disable:

sudo systemctl disable systemd-resolved.service

I updated the Ubuntu report with suggestions. [/update] Add: Note: the bug report : https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317 has a patch for 17.04 for some issues. Please check the bug report and if possible test the patch. Thank you!

[update]

Please check the above mentioned bug report, the issue seems to be resolved for 17.10 and with a simple command DNS leakage can be disabled too.

[/update]

DNS – How to Change systemd’s 127.0.0.53 Permanently

You can install a package resolvconf, which will modify the way /etc/resolv.conf is built up at system boot.

sudo apt install resolvconf

You can then create or modify a file /etc/resolvconf/resolv.conf.d/tail. If you put in this file a line nameserver 8.8.8.8, this line will be added at the end of /run/resolvconf/resolv.conf at boot. /etc/resolv.conf will now be a symbolic link to this file.

Post Scriptum:

Almost two years after posting my answer I came across https://bugs.launchpad.net/ubuntu/+source/ppp/+bug/1778946 which explains exactly why merely installing resolvconf solved a dns problem I had at the time. I feel I have to share this here.

Although my answer addresses the question in a proper way, indeed a warning should be added that, if you want/need to do this, probably something else is wrong. This was already stated by @intelfx at the time, 127.0.0.53 should work by itself.

According that bug report, after a pptp VPN goes down, resolv.conf is restored with the wrong access rights. ping ubuntu.com does not work, sudo ping ubuntu.com does. Installing resolvconf solved it, because it takes over resolv.conf, restoring it with correct rights. Changing systemd-resolve settings is no solution in this case, since the bug is in ppp. But an alternative, maybe simpler solution is sudo chmod a+r /etc/resolv.conf after VPN down. And this can be automated by putting an executable script in /etc/NetworkManager/dispatcher.d with contents:

#!/bin/sh
if [[ "$1"="ppp0" && "$2"="vpn-down" ]]; then  
    /bin/chmod a+r /etc/resolv.conf
fi

In all cases the contents of resolv.conf do not change. And, yes, I know pptp must be avoided because of security issues, but at the time I thought of it as a good excercise for an ubuntu newbie. I imagined it would work out of the box. Little did I know that it would give me a headache, as diagnosed so well by @intelfx.

Best Answer

Related Solutions

DNS – 16.10 Fails to Resolve

DNS – How to Change systemd’s 127.0.0.53 Permanently

Related Question