Ubuntu – NetworkManager-configured dnsmasq failing to forward requests

dnsdnsmasqnetworkmanagerresolvconfUbuntu

I use Lubuntu 17.04, on which the NetworkManager service (subsystem?) performs some configuration actions when I connect to a new network. One of these is (re)setting the /etc/resolv.conf file to:

# Generated by NetworkManager
search some-local-domain-here
nameserver 127.0.1.1

at least that's what I get when DHCP is used. There's a dnsmasq instance which listens on the local host (more information about it provided below.)

Now, on some networks, and pretty consistently, names cannot be resolved in this state of affairs; but if I replace the nameserver line with the actual nameserver's address I got from the DHCP lease (/var/lib/NetworkManager/dhclient-blah-blah.wlan0.lease) – resolution works fine.

Questions:

  • What could be the cause of this problem?
  • How can I force the NetworkManager-configured dnsmasq to actually forward name resolution requests to the DHCP-provided DNS server? (Note that it sometimes does this already – when this problem doesn't manifest)

Additional information:

The local DNS server is a dnsmasq instance, with the following command line:

/usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --cache-size=0 --clear-on-reload --conf-file=/dev/null --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d

The configuration directory /etc/NetworkManager/dnsmasq.d has a single file, whose contents are:

# Tell any system-wide dnsmasq instance to make sure to bind to interfaces
# instead of listening on 0.0.0.0
# WARNING: changes to this file will get lost if network-manager is removed.
bind-interfaces

Best Answer

Most modern Debian-related distributions tend to now use the /sbin/resolvconf tool to configure /etc/resolv.conf instead of editing it directly. This tool will keep track of DNS server information received for each interface, and it will prioritize the DNS server settings according to the order specified in file /etc/resolvconf/interface-order.

This way, you can get one set of DNS server settings when switching on Wi-Fi, another set when plugging in a network cable, a third one when starting up a VPN connection... and not end up with a messed-up configuration when you shut these down in a not-strictly-reverse order. Also, in this situation you can get two different mechanisms (e.g. NetworkManager and dnsmasq) vying for the control of the real /etc/resolv.conf, which might be

However, resolvconf is an optional package. If /sbin/resolvconf does not exist on your system, your /etc/resolv.conf might be under direct control of NetworkManager. But it looks like most of the scripting in the dnsmasq package is written assuming that /sbin/resolvconf exists - so without it you're relying on fallbacks which might not cover all situations adequately. You might want to try install it with apt-get install resolvconf or something similar and see how the DNS settings behave with it installed.

If you don't have the resolvconf package installed, I'm afraid NetworkManager does it all on its own - and in that case troubleshooting would require reading the source code of NetworkManager to see how it actually decides which DNS settings to use. resolvconf is far less opaque.

After dnsmasq starts up on (L)Ubuntu Zesty (17.04), its start-up scripts will run this command (assuming that the resolvconf package is installed):

echo "nameserver 127.0.0.1" | /sbin/resolvconf -a lo.dnsmasq

This tells /sbin/resolvconf that a new DNS name server has become available on interface lo, and it will update the real /etc/resolv.conf.

While this (or any) update of /etc/resolv.conf is happening, /sbin/resolvconf will run all scripts located in /etc/resolvconf/update.d. One of them is /etc/resolvconf/update.d/dnsmasq which is supposed to take up a list of all non-local DNS servers known to resolvconf at the time, and push them to /run/dnsmasq/resolv.conf, which is apparently the DNS server list that dnsmasq should be using.

Once every second, dnsmasq is supposed to check the modification time of /run/dnsmasq/resolv.conf and update its configuration.

However, your dnsmasq command line indicates that you may have made some changes to the default settings: you might want to replace the dnsmasq option --no-resolv with --resolv-file=/run/dnsmasq/resolv.conf... or just remove any NetworkManager or resolv.conf-related customizations, since it looks like the default settings of dnsmasq have been designed to integrate with NetworkManager and resolvconf in an appropriate way.

If the resolvconf tool is in use, you'll find the DNS server settings supplied by all the various possible sources in directory /run/resolvconf/interface/. The contents of each file will be similar to the real /etc/resolv.conf. The name of the file will refer to the source of the DNS server information, whether it is a static or DHCP configuration of a particular network interface, or NetworkManager in general, or the start-up scripts of a local DNS server.

Earlier, I mentioned a command line that will be executed at dnsmasq start-up if /sbin/resolvconf is installed. If you want to temporarily undo that (to test if DNS works if the local dnsmasq is bypassed), you can do it with this command:

resolvconf -d lo.dnsmasq

When you do that, the DNS settings of your highest-priority active network interface should take over in the real /etc/resolv.conf.

If you want to make dnsmasq startup scripts stop replacing the active DNS settings with the "nameserver 127.0.0.1" line (whether for troubleshooting or for other reasons), apparently adding DNSMASQ_EXCEPT=lo to /etc/default/dnsmasq will achieve that.

But since your nameserver line has value 127.0.1.1 instead, it is probably directly written by NetworkManager, and I have no clue how to fix that if resolvconf is out of the picture.

Related Question