I need to setup full-backup and diff backup. Can I do it on Server C?
You can do a copy only full, but unless C is the primary you cannot do a differential database backup.
Running below query on server C but it throwing 1 which means I can't take backup on that server.
A return value of 1 means it is "preferred" but doesn't necessarily mean you can run any type of backup on it.
1)How to revert back the AOAGs to original Synchronous mode -- Should I Perform a Forced Manual Failover of an Availability Group to PRODUCTION Servers(Sync) from DR/BCP (Asynchronous) node ?
You've failed over to asynchronous nodes. This means all of the database flows are paused and there is no current way (assuming it's a true disaster) how far behind your secondary replicas were. Now that they've come back up, we know that it isn't going to be 100% the same data (they are asynchronous). This was not mentioned in the question but I'm going to add it to the answer as it's extremely important as it's part of your SLA.
- Take a database backup of the original primary databases. To do this you will need to kick them out of the AG. Alternatively you could take a database snapshot (inside of SQL, not SAN).
- Resume data movement between the replicas at the database level.
- Set the local primary replica to synchronous and a remote replica to synchronous.
- Wait for them both to say 'Synchronized'. They will start out as 'Synchronizing'. This could take quite a while depending on multiple factors such as downtime, data generated, IO, Networking, etc.
- Once Synchronized, find a good time to take a minute or less outage. Use that time to do a planned manual failover.
- Set the local servers to be synchronous to each other.
- Remove synchronous from the remote replica and set it back to async.
2)We have one extra BCP node MSDTC and AOAGs Listener. Should I turn then online before I Perform a Forced Manual Failover of an Availability Group from PRODUCTION Servers(Sync) to DR/BCP (Asynchronous) node ?
How do you have an "extra AOAG listener" just chilling out? I don't understand that part of your question.
MSDTC is for distributed transactions, which aren't supported in 2012/2014 and only supported with certain restrictions (as of this moment) in 2016. Thus this is not required in a supported scenario. If you're going unsupported then this is still a moot point as the local MSDTC will be used.
Unless I'm missing something (completely possible) or not understanding, this is not needed.
Best Answer
You should first read How It Works: Always On–When Is My Secondary Failover Ready?. This would tell you about scenarios when your secondary is not failover ready and what could lead to "no automatic" failover
The data loss happened because you forced failover over to replica which was not synchronized. Look at the word "FORCE_FAILOVER_ALLOW_DATA_LOSS" it says itself that if you use me you "might" face data loss. What this command does is this beings online database discarding any transaction log blocks which originated on primary but somehow did not made on secondary and since secondary has no information about it it cannot replay it and hence you might face data loss when database comes online.
Let us call it transaction log block not data. This log block was on primary but due to unexpected shutdown it was not delivered to secondary and secondary was not synchronized. If you are able to bring primary online after some time without doing any forced failover you might have been able to save data loss but since you forced failover it is not possible. May be later if you bring primary replica online and the information is there in transaction log you can get it in primary replica. Shrinking has NO business here at all.