Systemd – Configure systemd-resolved and systemd-networkd for Local and Remote DNS

dnslinuxsystemdsystemd-networkdsystemd-resolved

I'm connected to local area network with access to the Internet through gateway. There is DNS server in local network which is capable of resolving hostnames of computers from local network.

I would like to configure systemd-resolved and systemd-networkd so that lookup requests for local hostnames would be directed (routed) exclusively to local DNS server and lookup requests for all other hostnames would be directed exclusively to another, remote DNS server.

Let's assume I don't know where the configuration files are or whether I should add more files and require their path(s) to be specified in the answer.

Best Answer

In the configuration file for local network interface (a file matching the name pattern /etc/systemd/network/*.network) we have to either specify we want to obtain local DNS server address from DHCP server using DHCP= option:

[Network]
DHCP=yes

or specify its address explicitly using DNS= option:

[Network]
DNS=10.0.0.1

In addition we need to specify (in the same section) local domains using Domains= option

Domains=domainA.example domainB.example ~example

We specify local domains domainA.example domainB.example to get the following behavior (from systemd-resolved.service, systemd-resolved man page):

Lookups for a hostname ending in one of the per-interface domains are exclusively routed to the matching interfaces.

This way hostX.domainA.example will be resolved exclusively by our local DNS server.

We specify with ~example that all domains ending in example are to be treated as route-only domains to get the following behavior (from description of this commit) :

DNS servers which have route-only domains should only be used for the specified domains.

This way hostY.on.the.internet will be resolved exclusively by our global, remote DNS server.

Note

Ideally, when using DHCP protocol, local domain names should be obtained from DHCP server instead of being specified explicitly in configuration file of network interface above. See UseDomains= option. However there are still outstanding issues with this feature – see systemd-networkd DHCP search domains option issue.

We need to specify remote DNS server as our global, system-wide DNS server. We can do this in /etc/systemd/resolved.conf file:

[Resolve]
DNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844

Don't forget to reload configuration and to restart services:

$ sudo systemctl daemon-reload
$ sudo systemctl restart systemd-networkd
$ sudo systemctl restart systemd-resolved

Caution!

Above guarantees apply only when names are being resolved by systemd-resolved – see man page for nss-resolve, libnss_resolve.so.2 and man page for systemd-resolved.service, systemd-resolved.

Related Solutions

Systemd-resolved block a domain name

Adding an entry to /etc/hosts is supposed to work and in my tests it worked as expected. My tests are on Fedora Rawhide, with version systemd-239-9.git9f3aed1.fc30.x86_64, so that's a pretty recent snapshot of systemd, maybe older versions will not work the same as expected...

Before adding the entry to /etc/hosts:

1) resolvectl query:

$ resolvectl query google.com
google.com: 172.217.6.78

-- Information acquired via protocol DNS in 1.4ms.
-- Data is authenticated: no

2) ping:

$ ping -c1 google.com
PING google.com (172.217.6.78) 56(84) bytes of data.
64 bytes from sfo07s17-in-f78.1e100.net (172.217.6.78): icmp_seq=1 ttl=54 time=12.4 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.435/12.435/12.435/0.000 ms

3) curl:

$ curl http://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

After adding the entry, in which case /etc/hosts looks like this:

$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 google.com
::1 google.com

Tests showed the blocking is working:

1) resolvectl query:

$ resolvectl query google.com
google.com: 127.0.0.1
            ::1

-- Information acquired via protocol DNS in 1.8ms.
-- Data is authenticated: yes

2) ping:

$ ping -c1 google.com
PING google.com(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.613 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.613/0.613/0.613/0.000 ms

3) curl:

$ curl http://google.com
curl: (7) Failed to connect to google.com port 80: Connection refused

So the block seems to be working.

I expected this would work, since this was brought up recently in an issue filed against systemd. Issue #9718 talked about adding millions of entries to /etc/hosts, which has one usecase and that's blacklisting domains, such as here.

Please note that there are quite a few moving parts here, so it's important to consider those while troubleshooting this.

My /etc/systemd/resolved.conf has no overridden configuration, all entries are commented out, network setup is using systemd-networkd with DHCP and no overrides either.

Output of resolvectl status includes:

Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no
Fallback DNS Servers: 8.8.8.8
                      8.8.4.4
                      2001:4860:4860::8888
                      2001:4860:4860::8844

Link 2 (ens33)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no

Configuration of /etc/resolv.conf is using the stub resolver:

$ ls -l /etc/resolv.conf
lrwxrwxrwx. 1 root root 39 Nov  7 22:08 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ grep '^[^#]' /etc/resolv.conf
nameserver 127.0.0.53

And nsswitch.conf is configured to use nss-resolve(8) according to the recommendation from its man page:

$ grep ^hosts: /etc/nsswitch.conf
hosts:      files resolve [!UNAVAIL=return] dns myhostname

If you still can't make it work, you might want to check these settings in your system and confirm they are all configured correctly. Or, at least, post your current configuration here (together with Linux distro and systemd version) to help diagnose why it might not be working for you.

How single-label dns lookup requests are handled by systemd-resolved

This issue gets very long to explain. The short (imprecise) description is:

where will the single-label lookup requests go?

Single label? (not localhost et al.): Always to the LLMNR system.

Multi-label?: To the DNS servers of each interface. On failure (or if not configured), to the global DNS servers.

Yes, the general sequence goes as described in systemd-resolved.service(8) BUT:

Routing of lookups may be influenced by configuring per-interface domain names. See systemd.network(5) for details.

Sets the systemd.network(5) as an additional resource for DNS resolution.

And, understand that From RFC 4795:

Since LLMNR only operates on the local link, it cannot be considered a substitute for DNS.

The sequence (simplified) is:

The local, configured hostname is resolved to all locally configured IP addresses ordered by their scope, or — if none are configured — the IPv4 address 127.0.0.2 (which is on the local loopback) and the IPv6 address ::1 (which is the local host).
The hostnames "localhost" and "localhost.localdomain" (and any hostname ending in ".localhost" or ".localhost.localdomain") are resolved to the IP addresses 127.0.0.1 and ::1.
The hostname "_gateway" is resolved to …
The mappings defined in /etc/hosts are included (forth and back).
If the name to search has no dots (a name like home. has a dot) it is resolved by the LLMNR protocol.

LLMNR queries are sent to and received on port 5355. RFC 4795
Multi word (one dot or more) names for some domain suffixes (like ".local", see full list with systemd-resolve --status) are resolved via MulticastDNS protocol.
Multi word names are checked against the Domains= list in systemd.network(5) per interface and if matched, the list of DNS servers of that interface are used.
Other multi-label names are routed to all local interfaces that have a DNS server configured, plus the globally configured DNS server if there is one.

Edit

The title of your question reads:

How single-label dns lookup requests are handled by systemd-resolved?

So, I centered my answer on systemd-resolved exclusively.

Now you ask:

If an interface is configured with search domain "mydomain" and LLMNR disabled, will any single-label lookup request be routed into this interface?
If an interface is configured with search domain "mydomain" and LLMNR enabled and a lookup request for "xyz" comes in, will "xyz" through LLMNR and "xyz.mydomain" throuth specified dns server both happen?

Those appear to be outside of systemd-resolved exclusively.

Let's try to analyze them:

LLMNR disabled ? How? Might I ask?. By disabling systemd-resolved itself with something similar to systemctl mask systemd-resolved?

If systemd-resolved is disabled/stopped there is no LLMNR in use (most probably, unless you install Avahi, Apple bonjour or a similar program) But certainly, that is outside of systemd-resolved configuration.

In this case, we should ask: what happens when a name resolution fails? (as there is no server to answer it). That is configured in nsswitch (file /etc/nsswitch.conf). The default configuration for Ubuntu (as Debian) contains this line:

hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname

That means (in nsswitch parlance):
1. Begin by checking the /etc/hosts file. If not found, continue.
2. Try mdns4_minimal (Avahi et al.) which will attempt to resolve the name via multicast DNS only if it ends with .local. If it does but no such mDNS host is located, mdns4_minimal will return NOTFOUND. The default name service switch response to NOTFOUND would be to try the next listed service, but the [NOTFOUND=return] entry overrides that and stops the search with the name unresolved. If mdns4_minimal return UNAVAIL (not running) then go to dns.
The plot thickens everybody wants to be first on the list to resolve names and everyone offers to do all resolutions by themselves.
1. The dns entry in nsswitch actually calls nss-resolve first which replace nss-dns
  
  nss-resolve is a plug-in module for the GNU Name Service Switch (NSS) functionality of the GNU C Library (glibc) enabling it to resolve host names via the systemd-resolved(8) local network name resolution service. It replaces the nss-dns plug-in module that traditionally resolves hostnames via DNS.
  
  Which will depend on the several DOMAINS= entries in general /etc/systemd/resolved.conf and/or each interface via /etc/systemd/network files. That was explained above the EDIT entry.
  
  Understand that sytemd-resolved might query the dns servers by itself before the dns entry in nsswitch.
2. If not found yet (without a [notfound=return] entry) then try the DNS servers. This will happen more-or-less immediately if the name does not end in .local, or not at all if it does. If you remove the [NOTFOUND=return] entry, nsswitch would try to locate unresolved .local hosts via unicast DNS. This would generally be a bad thing , as it would send many such requests to Internet DNS servers that would never resolve them. Apparently, that happens a lot.
3. The final myhostname acts as a last-resort resolver for localhost, hostname, *.local and some other basic names.
If systemd-resolved has a LLMNR=no set in /etc/systemd/resolved.conf the same list as above apply but systemd-resolved is still able to resolve localhost and to apply DOMAINS= settings (global or per interface).

Understand that There's the LLMNR setting in systemd-resolved and there's also the per-link LLMNR setting in systemd-networkd. link.

What does all that mean?

That it is very difficult to say with any certainty what will happen unless the configuration is very very specific. you will have to disable services and try (in your computer with your configuration) what will happen.

Q1

If an interface is configured with search domain "mydomain" and LLMNR disabled, will any single-label lookup request be routed into this interface?

Yes, of course, it may. That LLMNR is disabled only blocks local resolution (no other servers on the local (yes: .local) network will be asked) but the resolution for that name must find an answer (even if negative) so it may (if there is no NOTFOUND=return entry, for example) happen that the DNS servers for a matching interface are contacted to resolve mylocalhost.mylocaldomain when a resolution for mylocalhost was started and there is an entry for mylocaldomain in the "search domain". to answer in a general sense is almost imposible, too many variables.

Q2

If an interface is configured with search domain "mydomain" and LLMNR enabled and a lookup request for "xyz" comes in, will "xyz" through LLMNR and "xyz.mydomain" throuth specified dns server both happen?

No. if all is correctly configured a single label name "xyz" should only be resolved by LLMNR, and, even if asked, a DNS server should not try to resolve it. Well, that's theory. But the DNS system must resolve com (clearly, or the network would fall down as it is now). But there is a simple workaround: ask for com., it has a dot, it is a FQDN. In any case, a DNS server shuld answer with NOERROR (with an empty A (or AAAA)) if the server doesn't have enough information about a label and the resolution should continue with the root servers (for .). Or with a NXDOMAIN (the best answer to avoid further resolutions) for domains it knows that do not exist.

The only safe way to control this is to have a local DNS server and choose which names to resolve and which not to.