Ubuntu – 18.04 server – systemd-resolve returns cached cname NODATA for A lookup

dnsnetworkingserversystemd-resolved

Summary: DNS A record lookups fail to resolve due to cached CNAME NODATA lookups.

Details: Mail log reports errors with DNS lookups: 

 Host or domain name not found. Name service error for name=google.com type=A: Host found but no data record of requested type

After enabling resolve debugging, I see that dns lookups first query the CNAME record for the domain. This is often the root domain which has no CNAME and the lookup correctly returns NODATA.

However, when subsequent A lookups are performed, the NODATA result for the CNAME lookup is returned from the cache instead of doing an A lookup.

I can recreate this consistently by issuing the following commands:

~$ dig google.com CNAME
~$ dig google.com A

Here are the results from the debug log:

Aug 08 11:09:04 leopard systemd-resolved[555]: Transaction 17304 for <domain.com IN CNAME> scope dns on eth0/*.
Aug 08 11:09:04 leopard systemd-resolved[555]: Using feature level UDP+EDNS0 for transaction 17304.
Aug 08 11:09:04 leopard systemd-resolved[555]: Using DNS server 8.8.8.8 for transaction 17304.
Aug 08 11:09:04 leopard systemd-resolved[555]: Sending query packet with id 17304.
Aug 08 11:09:04 leopard systemd-resolved[555]: Processing query...
Aug 08 11:09:04 leopard systemd-resolved[555]: Processing incoming packet on transaction 17304. (rcode=SUCCESS)
Aug 08 11:09:04 leopard systemd-resolved[555]: Added NODATA cache entry for google.com IN CNAME 1799s
Aug 08 11:09:04 leopard systemd-resolved[555]: Transaction 17304 for <google.com IN CNAME> on scope dns on eth0/* now complete with <success> from network (unsigned).
Aug 08 11:09:04 leopard systemd-resolved[555]: Sending response packet with id 22860 on interface 1/AF_INET.
Aug 08 11:09:04 leopard systemd-resolved[555]: Freeing transaction 17304.

Results of the A record look-up:

Aug 08 11:09:37 leopard systemd-resolved[555]: Processing query...
Aug 08 11:09:51 leopard systemd-resolved[555]: Got DNS stub UDP query packet for id 3119
Aug 08 11:09:51 leopard systemd-resolved[555]: Looking up RR for google.com IN A.
Aug 08 11:09:51 leopard systemd-resolved[555]: NODATA cache hit for google.com IN A
Aug 08 11:09:51 leopard systemd-resolved[555]: Transaction 45189 for <google.com IN A> on scope dns on eth0/* now complete with <success> from cache (unsigned).
Aug 08 11:09:51 leopard systemd-resolved[555]: Freeing transaction 45189.
Aug 08 11:09:51 leopard systemd-resolved[555]: Sending response packet with id 3119 on interface 1/AF_INET.

Further Info: the server is running a LEMP stack. It appears that nginx does a DNS lookup before every request, and it starts with a CNAME lookup, then A lookup, then AAAA lookup. This causes the CNAME NODATA to be cached. Subsequently, when the mail server tries to send mail, it gets the cached NODATA record from resolve, leading to the error above.

Questions: is this expected behavoir (CNAME returned for A lookup)? Is there some configuration I can change to prevent the cached CNAME look-ups being returned for the A look-ups?

Diagnostic info:

~$ ip route
default via 85.159.215.1 dev eth0 proto static 
85.159.215.0/24 dev eth0 proto kernel scope link src 85.159.215.159 


~$ sudo systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4


~$ cat /etc/resolv.conf
nameserver 127.0.0.53

Best Answer

This seems like a real problem on the version of systemd Ubuntu 18.04 is using at this moment. https://github.com/systemd/systemd/issues/9833 and also on launchpad https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1818527

Not sure if Ubuntu will upgrade systemd version though.

Related Question