Sql-server – Configure Always On to use secondary site as DR

1. Yes the DR nodes need to be joined to the existing cluster. SQL Server 2016 has Distributed Availability Groups, which would a cluster at DR separate from the existing one, but you are on SQL Server 2014. The DR nodes should be able to see the fileshare witness. You can adjust quorum votes so that the DR nodes don't have a say in if quorum is down. I do add a fileshare in DR for when we failover to it, but it's not part of the cluster until the failover occurs.
2. The connection string would specify the AG listener and would not specify the two IPs. MultiSubnetFailover=True takes care of it by connecting to the IP that returned success first.
3. Yes it is very reasonable to swap roles. The last 3 companies I worked for had DR sites that we'd failover to 1-2 times per year, running production at DR for a few days and up to 3 weeks. Failovers are an offline event. They take about 15 seconds on AGs.

Based on the updated information, it looks like something ran the server out of CPU. I say something, because it could be inside of SQL Server - such as a bad parallel plan (or multiple calls of it), incorrect MaxDOP, etc. - or outside of SQL Server, we don't know from just the log what processes were taking up the most CPU.

The actual issue didn't tell us too much except that the lease was expected to be renewed about 10 seconds earlier:

2018/03/12-11:01:10.151 ERR [RES] SQL Server Availability Group : [hadrag] The lease is expired. The lease should have been renewed by 2018/03/12-16:01:00.068

The reason it is most likely (but not 100%, because if this is a VM it could have been over subscribed, paused for a snapshot, paused from migration, etc.) is due to the information later in the cluster log:

Date Time CPU%

---

3/12/2018 16:0:21.0, 95.167191
3/12/2018 16:0:31.0, 98.749704
3/12/2018 16:0:41.0, 91.797399
3/12/2018 16:0:51.0, 72.669750
3/12/2018 16:1:1.0, 62.595431

If you look, the lease timed out at 16:01:00 and the CPU usage was 92-99% the previous 30 seconds which just so happens to line up with the default lease timeout setting.

Now what?

This was an upgrade from 2016 to 2017 AND and edition upgrade from standard to Enterprise. I'd start by investigating things like MAXDOP, database compatibility level (for the optimizer and plans), and watching to see if any queries are going parallel at eating up CPU time. My guess is it has something to do with that, otherwise it's something outside of SQL Server causing the high CPU (which you can watch in SQL Server through the ring buffer).