DNSSEC – Going All-In on DNS Security Extensions

After more research, the problem appears to be the following:

The initial setup contained both forwarders that were DNSSEC-capable (GoogleDNS 8.8.8.8, 8.8.4.4) and some that weren't (OpenDNS 208.67.222.222, 208.67.220.220). I had BIND9 running with DNSSEC fully enabled, as per the following configuration:

dnssec-enable      yes;
dnssec-validation  yes;
dnssec-lookaside   auto;

a) Whenever a request (A?) was forwarded to the GoogleDNS servers, my_server got a reply (A), sent a DNSSEC-query (DS?) and got a proper answer. Resolution done, signature confirmed, all working like a charm, case closed (see above, the normal case).

b) Whenever a request (A?) was forwarded to the OpenDNS servers, my_server got a reply (A), sent a DNSSEC-query (DS?) which OpenDNS failed to answer properly as it does not support DNSSEC. So BIND9 threw an error in syslog, stating that it got insecure response and tried to get it's DNSSEC-validation elsewhere (see above, the problematic case).

I still don't fully understand what happened then, but obviously that's when the hiccups started. Now I don't know if the GoogleDNS-servers didn't like giving out DS?-answers without having served the corresponding A?-query first. But in both of the problematic cases (sueddeutsche.net, dasoertliche.de) it seems that the entries were also not properly signed so DNSSEC didn't produce correct validation. So DNSSEC-lookaside validation (DLV) was started (Type32769?) and again it all went south. No idea why.

c) Solution: After all this, I have done the following and have not yet run into any problem (so it seems the question is solved):

First, I switched forwarders to only

forwarders         { 8.8.8.8; 8.8.4.4; };

so that there's no longer a mixup of DNSSEC support. Second, I commented out

//dnssec-lookaside   auto;

because after digging through lots of tcpdumps, it appears that whenever resolution is slow, the delays are either caused by GoogleDNS taking some time to give back an answer (very rarely happens) or - regularly - happen during DLV. Since DLV is currently being phased out anyway with entries no longer available in 2017

https://www.isc.org/blogs/dlv/

I believe that's acceptable from a security-standpoint.

Now an alternative solution would be to ditch the GoogleDNS servers and only use OpenDNS as forwarders. But then I'd have to comment out all dnssec-entries mentioned above completely disabling DNSSEC, because OpenDNS does not support it. That would leave my dns queries open to attacks which would require adding an alternative layer of security like dnscrypt (as mentioned by Rui F Ribeiro). While that looks like a worthwile project (as it completely encrypts dns traffic, therefore not only keeping it unaltered but also unreadable by attackers), it's a little over my current time budget.

Any DNS experts out there who want to chime in if the explanations above make sense?

So, changing my wired eth0 interface to be managed solved this issue for me.

Changing ifupdown to managed=true in /etc/NetworkManager/NetworkManager.conf

[ifupdown]
managed=true

Then restart NetworkManager

sudo systemctl restart NetworkManager

After this it works flawlessly..

This was not 100%. I also applied theses changes to try and kill resolver

sudo service resolvconf disable-updates
sudo update-rc.d resolvconf disable
sudo service resolvconf stop

Big thanks to this blog post regarding the subject: https://ohthehugemanatee.org/blog/2018/01/25/my-war-on-systemd-resolved/

Lets pray this works.. This whole systemd-resolve business is just so ugly.