Sql-server – Failover timeout problem

clusteringfailoversql-server-2008-r2windows-server

I'm looking for an additional bit of help with regard to some problems I'm looking at for one of my company's clients. Basically I have two related problems with regard to a 2-node active/active cluster hosting an instance of 2008 R2 on each instance.

One of the instances, normally on node#2, is at RTM. And when failing over to node #1 there is a time-out bringing the SQL Server service on-line, however once the time-out has, er, 'timed out', the service can actually be brought on-line OK manually. And looking at the Log there are errors of the form

‘Login failed ….. Reason: Server is in script upgrade mode…..’.

At first I thought this was the result of a failed attempt at installing SP1 however I'm now not so sure. SP1 has definitely been installed on both nodes and the other instance, normally on node #1, is at SP1. I assume this was following the recommendation to install the service pack on the 'inactive' node, failing over and repeating the process. However I'm having difficulty interpreting the installation logs to see if just one instance has been updated or both, with one failing. So I was hoping someone could help me with that in terms of what log file I should be looking at.

Also, what might be the meaning of the 'script upgrade mode' errors? Does this indeed sound like a failed SP1 upgrade or is something else at play here? Curiously the problem only occurs in one direction – from node #2 to node #1. When failing back to node #2 the SQL Server service comes back on-line without the need for any manual intervention.

Best Answer

Had forgotten about updating this one after getting help from Microsoft Support. Turns out that the build number on one nodes was slightly different to the other due to a Security Patch that had only been applied to one node, possibly due to a Windows Update.

Related Solutions

SQL Server 2012 – Is Service Pack 2 Installed on Passive Node?

The logs led us to review our Cluster services:

There was an error to lookup cluster resources. Error: There was a failure to call cluster code from a provider. Exception message: Generic failure . Status code: 4104. Description: .
HResult : 0x86d80014
FacilityCode : 1752 (6d8)
ErrorCode : 20 (0014)
Data:
    errorMessage = There was a failure to call cluster code from a provider. Exception message: Generic failure . Status code: 4104. Description: .

We determined that the replication agent service was the cause of the incomplete installation.

We removed the service from Cluster Resources and re-applied SP2 without success. We stopped the service on the node and re-applied SP2 without success.

Error: Failed to run patch request for instance: MSSQLSERVER (exit code: -2032664552)

We uninstalled the service from the server nodes, and then successfully "re-applied" SP2.

Follow-up actions:

We'll be contacting the replication agent vendor to discuss why this was required.
Review the Microsoft document "Patching SQL Server Failover Cluster Instances with Cluster-Aware Updating (CAU)", available here, which I discovered today.

Sql-server – Quorum disk conundrum

You should not have the quorum disk as a resource inside a "Service or Application". It needs to be configured separately, and should be visible only under the "Disk Witness in Quorum" section in the "Storage" configuration pane.

When the cluster is configured with either "No Majority - Disk Only" or "Node and Disk Majority", the cluster service uses the quorum disk as a "vote" - since the disk can only reside at a single node. If the nodes cannot communicate with each, only a single node will be able to connect to the disk and obtain the required number of votes to achieve quorum. This prevents a partition where both nodes think they can run the services and applications configured in the cluster. If no nodes can connect to the quorum disk, none of the nodes will run the cluster services and applications.

Assigning the quorum disk to a Service or Application means the disk will go offline when a failover occurs. If the disk goes offline, it cannot be used as a vote, and the entire cluster will not operate.

See Windows Server 2008 R2 Cluster Docs and The process of achieving quorum for details

Related Question