MacOS Network – Website Unreachable / PR_END_OF_FILE_ERROR / Tries to Connect to Localhost Instead

Since some weeks I have a very strange issue, where I'm stuck how to further debug, specially as I'm more of a Linux Guy in front of a MacOS (10.15.4)

From time to time I get in Firefox/Chrome a error when requesting a website with the error PR_END_OF_FILE_ERROR (firefox) or ERR_CONNECTION_CLOSED (chrome). Waiting a couple of minutes solves the issue (tried to really do nothing – stepping away for 5min). Sometimes it then works for the next 3h without a problem, sometimes it comes back after 5min.

So its super inconsistence and I didn't found out yet, what invokes the issue.

What I found out so far:

• seems to be not an issue with antivirus Sophos (9.9.8) – disabled it, still had the issue
• seems to be not an issue with VPN – got the issue if I'm in the company VPN and also if I'm not connected to the VPN (OpenVPN over Tunnelblick or IPSec)
• get it if I'm connected with WiFi and when connected with Ethernet
• get it at home and get it at work
• sometimes some sites works, others don't, like MS Teams seems to work all the time (also happens if MS Teams is not running)

Last clue I have is that it seems to be something like a hidden proxy, because of the output from curl I get, if the issue persists.

1st try:

Tue May 19 08:31:12,xxx@MBP-133-xxx-2 ~ $ curl stackoverflow.com -v > /dev/null
Trying 151.101.129.69...
* TCP_NODELAY set
* Connected to stackoverflow.com (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: stackoverflow.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host stackoverflow.com left intact
curl: (52) Empty reply from server
* Closing connection 0

2nd try (just 1second later)

Tue May 19 08:31:13,xxx@MBP-133-xxx-2 ~ $ curl stackoverflow.com -v > /dev/null
Trying 151.101.129.69...
* TCP_NODELAY set
* Connected to stackoverflow.com (151.101.129.69) port 80 (#0)
> GET / HTTP/1.1
> Host: stackoverflow.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< cache-control: no-cache, no-store, must-revalidate
< location: https://stackoverflow.com/
< server: Microsoft-IIS/10.0
< x-flags: AA
< x-aspnet-duration-ms: 0
< x-request-guid: 450a82e6-54bc-483f-8825-a778ac09d170
< x-is-crawler: 1
< x-providence-cookie: 0fa36a94-4864-6d95-d737-cc8ab7e2a285
< Transfer-Encoding: chunked
< Accept-Ranges: bytes
< Date: Tue, 19 May 2020 06:31:14 GMT
< Via: 1.1 varnish
< Connection: keep-alive
< X-Served-By: cache-fra19134-FRA
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1589869874.022703,VS0,VE93
< Vary: Fastly-SSL
< X-DNS-Prefetch-Control: off
< Set-Cookie: prov=0fa36a94-4864-6d95-d737-cc8ab7e2a285; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
<
{ [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host stackoverflow.com left intact
* Closing connection 0

If I do this, then it more or less jumps between both states. Sometimes more failing, sometimes more succeeding.

But the biggest issue I see here for the failing connection is Trying 151.101.129.69... and then Connected to stackoverflow.com (127.0.0.1)

I do not have an proxy setup, in the /etc/hosts are only develop entries, definitiv not stackoverflow.com. Between both curls I didn't do anything (it was really just doing two calls in the row).

I'm open for any suggestion how to debug this further.

edit-00:

Requested curl --trace-ascii trace.log stackoverflow.com

Failure:

== Info:   Trying 151.101.193.69...
== Info: TCP_NODELAY set
== Info: Connected to stackoverflow.com (127.0.0.1) port 80 (#0)
=> Send header, 81 bytes (0x51)
0000: GET / HTTP/1.1
0010: Host: stackoverflow.com
0029: User-Agent: curl/7.64.1
0042: Accept: */*
004f:
== Info: Empty reply from server
== Info: Connection #0 to host stackoverflow.com left intact

Working:

== Info:   Trying 151.101.129.69...
== Info: TCP_NODELAY set
== Info: Connected to stackoverflow.com (151.101.129.69) port 80 (#0)
=> Send header, 81 bytes (0x51)
0000: GET / HTTP/1.1
0010: Host: stackoverflow.com
0029: User-Agent: curl/7.64.1
0042: Accept: */*
004f:
<= Recv header, 32 bytes (0x20)
0000: HTTP/1.1 301 Moved Permanently
<= Recv header, 52 bytes (0x34)
0000: cache-control: no-cache, no-store, must-revalidate
<= Recv header, 38 bytes (0x26)
0000: location: https://stackoverflow.com/
<= Recv header, 28 bytes (0x1c)
0000: server: Microsoft-IIS/10.0
<= Recv header, 13 bytes (0xd)
0000: x-flags: AA
<= Recv header, 25 bytes (0x19)
0000: x-aspnet-duration-ms: 0
<= Recv header, 54 bytes (0x36)
0000: x-request-guid: 6a97f488-95a4-4071-bcc0-f83d8209df2f
<= Recv header, 17 bytes (0x11)
0000: x-is-crawler: 1
<= Recv header, 59 bytes (0x3b)
0000: x-providence-cookie: 3108f7a8-ac68-c64f-680e-2e5dd3bcc55c
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 22 bytes (0x16)
0000: Accept-Ranges: bytes
<= Recv header, 37 bytes (0x25)
0000: Date: Tue, 19 May 2020 06:57:58 GMT
<= Recv header, 18 bytes (0x12)
0000: Via: 1.1 varnish
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 33 bytes (0x21)
0000: X-Served-By: cache-fra19171-FRA
<= Recv header, 15 bytes (0xf)
0000: X-Cache: MISS
<= Recv header, 17 bytes (0x11)
0000: X-Cache-Hits: 0
<= Recv header, 38 bytes (0x26)
0000: X-Timer: S1589871478.446474,VS0,VE94
<= Recv header, 18 bytes (0x12)
0000: Vary: Fastly-SSL
<= Recv header, 29 bytes (0x1d)
0000: X-DNS-Prefetch-Control: off
<= Recv header, 139 bytes (0x8b)
0000: Set-Cookie: prov=3108f7a8-ac68-c64f-680e-2e5dd3bcc55c; domain=.s
0040: tackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/;
0080:  HttpOnly
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 5 bytes (0x5)
0000: 0
0003:
== Info: Connection #0 to host stackoverflow.com left intact

Output scutil --dns

DNS configuration

resolver #1
  search domain[0] : company.com
  nameserver[0] : 172.27.10.42
  nameserver[1] : 172.27.10.41
  nameserver[2] : 172.27.10.43
  if_index : 10 (en14)
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : company.com
  nameserver[0] : 172.27.10.42
  nameserver[1] : 172.27.10.41
  nameserver[2] : 172.27.10.43
  if_index : 10 (en14)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

Content /etc/resolv.com

search company.com
nameserver 172.27.10.42
nameserver 172.27.10.41
nameserver 172.27.10.43

My /etc/hosts is long and has a lot of entries, because of local development. But it breaks down to this:

255.255.255.255 broadcasthost
::1             localhost
fe80::1%lo0     localhost
127.0.0.1       localhost

# a lot of entries like
127.0.0.1       abc.nauts.eu

127.0.0.1   localhost drupal8.local gitlab pharmadbv2.local webgrind.local
127.0.0.1   xhprof.nauts.eu
255.255.255.255 broadcasthost
::1             localhost

127.0.0.1   kubernetes.docker.internal

edit-01: It is a HTTP/HTTPs issue only. Like if this issue is happening for an internal server, I have no problem connecting to it via ssh.

edit-02

curl --noproxy company.com --trace-ascii trace2.log jenkins.company.com

== Info:   Trying 172.27.10.63...
== Info: TCP_NODELAY set
== Info: Connected to jenkins.company.com (127.0.0.1) port 80 (#0)
=> Send header, 84 bytes (0x54)
0000: GET / HTTP/1.1
0010: Host: jenkins.company.com
002c: User-Agent: curl/7.64.1
0045: Accept: */*
0052:
== Info: Empty reply from server
== Info: Connection #0 to host jenkins.company.com left intact