Linux – Error in docker pull (linux) “request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)”

dockerlinuxresolv.confubuntu 18.04

Issue type

— error

When trying to pull docker containers from I am confronted with this error:

Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

## OS version
Ubuntu 18.04 (bionic)
## Docker version
Client:
Version:           18.09.4
API version:       1.39
Go version:        go1.10.8
Git commit:        d14af54266
Built:             Wed Mar 27 18:35:44 2019
OS/Arch:           linux/amd64
Experimental:      false
Server: Docker Engine - Community
Engine:
Version:          18.09.4
API version:      1.39 (minimum version 1.12)
Go version:       go1.10.8
Git commit:       d14af54
Built:            Wed Mar 27 18:01:48 2019
OS/Arch:          linux/amd64
Experimental:     false

Steps to reproduce

docker pull hello-world

I am not behind a proxy or vpn (which seems to be a problem in other posts I've seen about this error)

I have attempted to search for a fix but to no avail. service docker restart does not resolve the issue. I have found that adding "nameserver 8.8.8.8" to etc/resolv.conf temporarily resolves the issue, but resolv.conf is rewritten/updated to the old version automatically, so this is not an ideal solution. I have been able to pull docker images from docker hub on this computer and on my current network in the past, so presumably some setting was changed, causing this issue. Any help would be much appreciated! Thanks very much.

.

If it is relevant, here is output from docker info:

Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 9
Server Version: 18.09.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1035-oem
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.05GiB
Name: ###(redacted)###
ID: ###(redacted)###
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support

Output from dig index.docker.io using 127.0.0.53 (default in resolv.conf)

 ; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> index.docker.io
 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57173
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags:; udp: 65494
 ;; QUESTION SECTION:
 ;index.docker.io.      IN  A

 ;; ANSWER SECTION:
index.docker.io.    300 IN  CNAME   elb-io.us-east-1.aws.dckr.io.
elb-io.us-east-1.aws.dckr.io. 899 IN    CNAME   us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com.
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 59 IN A 52.207.42.240
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 59 IN A 3.91.211.1
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 59 IN A 52.54.178.62

;; Query time: 404 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Apr 15 09:36:42 MDT 2019
;; MSG SIZE  rcvd: 212

Output from dig index.docker.io using 8.8.8.8

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> index.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37695
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;index.docker.io.       IN  A

;; ANSWER SECTION:
index.docker.io.    133 IN  CNAME   elb-io.us-east-1.aws.dckr.io.
elb-io.us-east-1.aws.dckr.io. 837 IN    CNAME   us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com.
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 56 IN A 52.207.42.240
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 56 IN A 3.91.211.1
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 56 IN A 52.54.178.62

;; Query time: 19 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Apr 15 09:49:22 MDT 2019
;; MSG SIZE  rcvd: 212

Best Answer

Likely not a problem with your Docker installation.

If changing the DNS to 8.8.8.8 resolves the issue then perhaps you get a different address from 8.8.8.8 than from your regular DNS service. You can use dig to get the address using both configurations:

dig index.docker.io

; <<>> DiG 9.10.3-P4-Ubuntu <<>> index.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28068
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;index.docker.io.               IN      A

;; ANSWER SECTION:
index.docker.io.        926     IN      CNAME   elb-io.us-east-1.aws.dckr.io.
elb-io.us-east-1.aws.dckr.io. 926 IN    CNAME   us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com.
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 926 IN A 52.54.178.62
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 926 IN A 3.91.211.1
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 926 IN A 52.207.42.240

;; Query time: 2 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Sun Apr 14 09:15:20 CEST 2019
;; MSG SIZE  rcvd: 214

If you don't want to change your DNS, you can still add one of the addresses above in your /etc/hosts, this won't be reset when you reboot. Of course this prevents you from using other servers in the cluster, and the specific address you pick can become invalid.

If you standard DNS in resolv.conf is 127.0.0.something, your system is likely set up to use a NetworkManager-controlled dsnmasq: DNS queries are handled by a service on your system, that forwards them to various DNS under the hood. You can find out the actual DNS used (DNS is set per network interface) using:

nmcli dev show | grep DNS

And then continue the quest for the slow/invalid DNS from there.