Networking – “reply from unexpected source” when DNS server is running as docker container in the docker host

dnsdockernetworking

In a machine with IP 192.168.100.6, I can run a docker container and perform dns without issues from within the container:

$ docker run --rm -it xenial-networking bash

root@255c2ffc38cb:/# dig registry.mynet

; <<>> DiG 9.10.3-P4-Ubuntu <<>> registry.mynet
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1540
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry.mynet.              IN      A

;; ANSWER SECTION:
registry.mynet.       0       IN      A       192.168.100.16

;; Query time: 0 msec
;; SERVER: 192.168.100.16#53(192.168.100.16)
;; WHEN: Thu May 17 07:54:28 UTC 2018
;; MSG SIZE  rcvd: 61

(as you can see, the registry.mynet is in the same host as the dns server, 192.168.100.16)

For reference, the resolver is configured as follows in the docker container:

root@255c2ffc38cb:/# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.100.16
nameserver 192.168.100.4
nameserver 192.168.100.3
nameserver 192.168.100.2
search openstacklocal

(which is a copy of the resolver config in the docker host), as per these rules

In the 192.168.100.16 machine, where the DNS server is actually running (as a docker service, see below for the compose config), the same does not work: although the resolver config is exactly the same: I get a "reply from unexpected source", and no name resolution:

root@e7de85671e86:/# dig registry.mynet
;; reply from unexpected source: 172.17.0.1#53, expected 192.168.100.16#53

; <<>> DiG 9.10.3-P4-Ubuntu <<>> registry.mynet
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 12820
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry.mynet.              IN      A

;; AUTHORITY SECTION:
.                       10800   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2018051700 1800 900 604800 86400

;; Query time: 97 msec
;; SERVER: 192.168.100.4#53(192.168.100.4)
;; WHEN: Thu May 17 07:58:25 UTC 2018
;; MSG SIZE  rcvd: 120

The resolver configuration for this container is the same:

root@e7de85671e86:/# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.100.16
nameserver 192.168.100.4
nameserver 192.168.100.3
nameserver 192.168.100.2
search openstacklocal

Why is the reply coming from 172.17.0.1.

Note

The compose configuration for the dns service is as follows:

  dnsmasq:
    image: andyshinn/dnsmasq:2.78
    volumes:
      - ./dnsmasq/conf/dnsmasq.conf:/etc/dnsmasq.conf
      - ./dnsmasq/conf/dnsmasq.d:/etc/dnsmasq.d
      - ./dnsmasq/conf/hosts:/etc/hosts
    network_mode: "host"
    cap_add:
      - NET_ADMIN
    restart: always
    command: --log-facility=- --log-queries=extra --all-servers --conf-file=/etc/dnsmasq.conf

Update

Changing the resolver in the docker host with problems (the 192.168.100.16 machine) to:

$ cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 172.17.0.1
nameserver 192.168.100.4
nameserver 192.168.100.3
nameserver 192.168.100.2
search openstacklocal

Gets rid of the problem. I still do no understand why 192.168.100.16 nameserver is not working properly in containers running in the 192.168.100.16 host (in the host itself it is working fine)

Best Answer

I had the same problem.

For everyone out there facing the same issue:

I'm working in Linux (specifically Ubuntu), I had dnsmasq setup in my host machine. Same setup works in mac, but in Linux, the container inside is not able to resolv to my local dnsmasq, even though I use the correct IP address to point to my machine (not localhost/127.0.0.1). Basically the same setup as above. It didn't work.

After I specifically use network: bridge in all my services,the same IP now connected. So, the problem is, default network in docker engine in Linux is not connected by default to host interfaces. We have to specifically use bridged network for this. Now, why does it work on Mac? I don't know. Probably because the host interfaces were bridged by default in Mac.

Related Question