SQL Server 2017 – Adding a Third Replica to AG Without Adding a Third Machine to WSFC

1. I have Windows Server Failover Cluster setup on 2012 R2 with SQL Server 2012, Can I have my new server with Windows Server 2012 R2 with SQL Server 2016?

Yes, you can... however, it will not be readable as a secondary and if you ever fail over to it, the SQL Server 2012 instance will no longer synchronize and your databases will be upgraded. Unless you're planning on doing an upgrade, I wouldn't mix and match versions - it gets you nothing.

2. Can I add it on my existing AlwaysOn Availability Group, which is configured with SQL Server 2012?

Yes, but do see above for restrictions and limitations. I would only do this if you're starting a rolling upgrade scenario and plan to finish it fairly quickly. Do not leave multiple versions in the same topology for any appreciable amount of time.

In a 2 node Availability Group (or mirror) a 3rd node is important if you want automatic HA. If your secondary cannot connect to the primary, the secondary has to work out if it can come online as the primary, or if the error is actually with the secondary. Without a 3rd party, this is impossible.

The File Share Witness will be simpler than adding another Windows node just to work as a witness. It can be anywhere, what is important is that both nodes will be able to see the witness if the other node goes offline. You might need to think extra hard if you have a virtual environment, you need to consider if the physical node goes offline will your fileshare still be available.

Additional thoughts for multi data centers

If the 2 nodes are in different data centers (you mentioned DR, so I'm assuming) then having a file share in the cloud may be helpful (you can use Azure Blob Storage as a witness). This means if one of your primary data center goes offline, SQL will automatically failover to the secondary. If you host the fileshare at your secondary and the secondary goes offline, then your primary server will go offline as well (can't contact the secondary or the file share so assumes that it is no longer primary).