Sql-server – Why does resource in AlwaysOn fail

availability-groupssql serversql server 2014

I have two SQL Servers (12.0.4100.1) and Availability Group (AG) configured on them. Sometimes some server hangs. It responds on ping, but RDP connection are not available, SQL Server doesn't respond and so on. After restarting, everything is ok. I am trying to find out why server hangs. I get cluster log on both nodes and there is following information:

Node1 (PRIMARY):

000016e0.00001808::2015/12/12-09:41:13.053 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:41:13.047
000016e0.00001808::2015/12/12-09:41:33.056 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:41:33.050
0000118c.000038e4::2015/12/12-09:42:11.969 INFO  [API] s_ApiGetQuorumResource final status 0.
0000118c.000038e4::2015/12/12-09:42:11.969 INFO  [API] s_ApiGetQuorumResource final status 0.
0000118c.000038e4::2015/12/12-09:42:11.969 WARN  [API] s_ApiOpenResourceEx: Resource  not found, status = 5007
000016e0.00001808::2015/12/12-09:43:03.067 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'resource' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:43:03.063
000016e0.00001808::2015/12/12-09:43:13.068 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'resource' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:43:13.063
0000118c.000053dc::2015/12/12-09:46:14.274 INFO  [API] s_ApiGetQuorumResource final status 0.
0000118c.000053dc::2015/12/12-09:46:14.274 INFO  [API] s_ApiGetQuorumResource final status 0.
0000118c.000053dc::2015/12/12-09:46:14.274 WARN  [API] s_ApiOpenResourceEx: Resource  not found, status = 5007
000016e0.00001808::2015/12/12-09:47:33.113 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:47:33.107
000016e0.00001808::2015/12/12-09:47:53.115 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:47:53.107
000016e0.00001808::2015/12/12-09:52:33.155 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'resource' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:52:33.150
000016e0.00001808::2015/12/12-09:52:43.157 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'resource' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:52:43.153
000016e0.00001808::2015/12/12-09:54:23.177 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:54:23.170
000016e0.00001808::2015/12/12-09:54:33.178 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:54:33.170
000016e0.00001808::2015/12/12-09:55:03.179 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'resource' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:55:03.173
000016e0.00001808::2015/12/12-09:55:13.181 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'resource' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:55:13.177
000016e0.00001808::2015/12/12-09:57:53.211 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'clean' to 'warning' at 2015-12-12 11:57:53.200
000016e0.00001808::2015/12/12-09:58:03.214 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'warning' to 'clean' at 2015-12-12 11:58:03.203
000016e0.00001808::2015/12/12-10:01:00.878 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'system' health state has been changed from 'clean' to 'warning' at 2015-12-12 12:01:00.877
000016e0.00001808::2015/12/12-10:01:00.887 INFO  [RES] SQL Server Availability Group: [hadrag] SQL Server component 'query_processing' health state has been changed from 'clean' to 'warning' at 2015-12-12 12:01:00.877
0000118c.00002ed0::2015/12/12-10:01:01.071 INFO  [RCM] rcm::RcmApi::FailResource: (AG)
0000118c.00004824::2015/12/12-10:01:01.078 INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'AG', gen(1) result 0.
0000118c.00004824::2015/12/12-10:01:01.078 INFO  [RCM] TransitionToState(AG) Online-->ProcessingFailure.
0000118c.00004824::2015/12/12-10:01:01.088 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (AG, Online --> Failed)
0000118c.00004824::2015/12/12-10:01:01.088 ERR   [RCM] rcm::RcmResource::HandleFailure: (AG)
0000118c.00004824::2015/12/12-10:01:01.100 INFO  [RCM] resource AG: failure count: 1, restartAction: 2.
0000118c.00004824::2015/12/12-10:01:01.100 INFO  [RCM] Will restart resource in 500 milliseconds.
0000118c.00004824::2015/12/12-10:01:01.100 INFO  [RCM] TransitionToState(AG) ProcessingFailure-->[WaitingToTerminate to DelayRestartingResource].
0000118c.00004824::2015/12/12-10:01:01.100 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (AG, Failed --> Pending)
0000118c.00004824::2015/12/12-10:01:01.100 INFO  [RCM] TransitionToState(AG) [WaitingToTerminate to DelayRestartingResource]-->[Terminating to DelayRestartingResource].
0000118c.00005110::2015/12/12-10:01:01.137 WARN  [RCM] AG cannot be hosted on node 2
0000118c.00005110::2015/12/12-10:01:01.137 WARN  [RCM] Possible owners: <vector len='1'>
0000118c.00005110::2015/12/12-10:01:01.137 WARN      <item>1</item>
0000118c.00005110::2015/12/12-10:01:01.137 WARN  </vector>
0000118c.00005110::2015/12/12-10:01:01.138 WARN  [RCM] AG cannot be hosted on node 3
0000118c.00005110::2015/12/12-10:01:01.138 WARN  [RCM] Possible owners: <vector len='1'>
0000118c.00005110::2015/12/12-10:01:01.138 WARN      <item>1</item>
0000118c.00005110::2015/12/12-10:01:01.138 WARN  </vector>
0000118c.00005110::2015/12/12-10:01:01.143 INFO  [RCM] rcm::RcmApi::AddPossibleOwner: (AG, 1)
000016e0.000017e8::2015/12/12-10:01:01.160 ERR   [RES] SQL Server Availability Group <AG>: [hadrag] Lease Thread terminated
000016e0.0000508c::2015/12/12-10:01:01.216 INFO  [RES] SQL Server Availability Group: [hadrag] Stopping Health Worker Thread
000016e0.0000181c::2015/12/12-10:01:01.216 INFO  [RES] SQL Server Availability Group: [hadrag] Health worker was asked to terminate
000016e0.00001808::2015/12/12-10:01:01.912 INFO  [RES] SQL Server Availability Group: [hadrag] SQLMoreResults() returns -1 with following information
000016e0.0000181c::2015/12/12-10:01:01.912 INFO  [RES] SQL Server Availability Group: [hadrag] Change diagnostics interval worker is stopped
000016e0.00001808::2015/12/12-10:01:01.912 ERR   [RES] SQL Server Availability Group: [hadrag] ODBC Error: [HY008] [Microsoft][SQL Server Native Client 11.0]Operation canceled (0)
000016e0.00001808::2015/12/12-10:01:01.912 ERR   [RES] SQL Server Availability Group: [hadrag] ODBC Error: [01000] [Microsoft][SQL Server Native Client 11.0][SQL Server]  (0)
000016e0.00001808::2015/12/12-10:01:01.912 INFO  [RES] SQL Server Availability Group: [hadrag] No more diagnostics results
000016e0.00001808::2015/12/12-10:01:01.912 INFO  [RES] SQL Server Availability Group: [hadrag] Diagnostics is stopped
000016e0.00001808::2015/12/12-10:01:01.921 INFO  [RES] SQL Server Availability Group: [hadrag] Disconnect from SQL Server
000016e0.00001808::2015/12/12-10:01:01.988 INFO  [RES] SQL Server Availability Group: [hadrag] Extended Event logging is stopped
000016e0.00001808::2015/12/12-10:01:02.020 INFO  [RES] SQL Server Availability Group: [hadrag] Extended Event target state:
000016e0.00001808::2015/12/12-10:01:02.020 INFO  [RES] SQL Server Availability Group: [hadrag] Extended Event session summary: dropped buffers = 0, dropped events = 0
000016e0.0000508c::2015/12/12-10:01:02.031 INFO  [RES] SQL Server Availability Group: [hadrag] Stopping Change Diagnostics interval Worker Thread

Node2 (SECONDARY) This node hangs:

00000da8.00000d4c::2015/12/12-08:03:34.402 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Re-registering DNS records time period (418259 secs ) between last registration and now is greater than 86400
00000da8.00000d4c::2015/12/12-08:03:34.530 INFO  [ClNet] Adapter Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿* 4 RFC2863 operational status = 1.
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet] Created adapter: DeviceGuid:     D3C156E4-342E-42BD-8180-48C855430C33
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  DeviceName:     Microsoft Failover Cluster Virtual Adapter
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  ConnectoidName: Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿* 4
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  Netbios/TCP:    1
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  DNS Suffix:
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::1%1
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::2%1
00000da8.00000d4c::2015/12/12-08:03:34.541 DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::3%1
00000da8.00000d4c::2015/12/12-08:03:34.542 INFO  [ClNet] Adapter Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿ 2 RFC2863 operational status = 1.
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet] Created adapter: DeviceGuid:     C1D9D5D3-0F70-49D2-B87F-2F141B7C577D
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DeviceName:     æÑΓÑó«Ñ »«ñ¬½ετÑ¡¿Ñ ß π߬«αÑ¡¿Ñ¼ óó«ñá-óδó«ñá Intel(R) PRO/1000 EB #2
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  ConnectoidName: Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿ 2
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  Netbios/TCP:    1
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DNS Suffix:     DOM.LOC
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      10.75.0.1
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      10.75.0.2
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      10.75.0.5
00000da8.00000d4c::2015/12/12-08:03:34.542 INFO  [ClNet] Adapter isatap.{D3C156E4-342E-42BD-8180-48C855430C33} RFC2863 operational status = 2.
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet] Created adapter: DeviceGuid:     592D38E8-2ED5-46A2-9E33-6DC413D3D1B6
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DeviceName:     Çñá»ΓÑα Microsoft ISATAP
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  ConnectoidName: isatap.{D3C156E4-342E-42BD-8180-48C855430C33}
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  Netbios/TCP:    0
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DNS Suffix:
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::1%1
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::2%1
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::3%1
00000da8.00000d4c::2015/12/12-08:03:34.542 INFO  [ClNet] Adapter isatap.DOM.LOC RFC2863 operational status = 2.
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet] Created adapter: DeviceGuid:     6E84C6A5-C033-4609-A4EF-082190C0C89E
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DeviceName:     Çñá»ΓÑα Microsoft ISATAP #2
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  ConnectoidName: isatap.DOM.LOC
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  Netbios/TCP:    0
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DNS Suffix:     DOM.LOC
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      10.75.0.1
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      10.75.0.2
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DnsServer:      10.75.0.5
00000da8.00000d4c::2015/12/12-08:03:34.542 INFO  [ClNet] Adapter Teredo Tunneling Pseudo-Interface RFC2863 operational status = 2.
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet] Created adapter: DeviceGuid:     792BFDE9-B449-4DF2-AF18-5EBA1E93DB04
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DeviceName:     Teredo Tunneling Pseudo-Interface
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  ConnectoidName: Teredo Tunneling Pseudo-Interface
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  Netbios/TCP:    0
00000da8.00000d4c::2015/12/12-08:03:34.542 DBG   [ClNet]                  DNS Suffix:
00000da8.00000d4c::2015/12/12-08:03:34.634 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: adapter Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿ 2 (3)
00000a4c.00001f28::2015/12/12-08:03:34.655 INFO  [GUM] Node 2: Processing RequestLock 2:227
00000a4c.00001f94::2015/12/12-08:03:34.967 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 6243)
00000da8.00000d4c::2015/12/12-08:03:35.686 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Trying to obtain the VSToken for Core Cluster Name resource
00000da8.00000d4c::2015/12/12-08:03:35.686 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: GetCoreNetnameObject_VSToken returning status 0
00000da8.00000d4c::2015/12/12-08:03:36.090 WARN  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Testing 81.0.75.10.in-addr.arpa. for dynamic updates over adapter 'Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿ 2' failed, status 9505.
00000da8.00000d4c::2015/12/12-08:03:36.098 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: DNS name VSQL flushed from DNS resolver Cache
00000da8.00000d4c::2015/12/12-08:03:36.104 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Replaced DNS name VSQL.dom.loc with IP Address 10.75.0.81 over adapter 'Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿ 2'.
00000da8.00000d4c::2015/12/12-08:03:36.105 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: DNS name VSQL.dom.loc Registration with LSA was successful
00000da8.00000d4c::2015/12/12-08:03:36.116 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Modified DNS name VSQL.dom.loc with IP Address 10.75.0.81 over adapter 'Å«ñ¬½ετÑ¡¿Ñ »« ½«¬á½∞¡«⌐ ßÑΓ¿ 2'.
00000da8.00000d4c::2015/12/12-08:03:36.117 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: DNS name VSQL.dom.loc Registration with LSA was successful
00000da8.00000d4c::2015/12/12-08:03:36.660 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Trying new logon (last 596216000, current 682618500)
00000da8.00000d4c::2015/12/12-08:03:36.892 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: Got new Logon Session.
00000da8.00000d4c::2015/12/12-08:03:36.893 INFO  [RES] Network Name <ê¼∩ ¬½áßΓÑαá>: TimerQueueTimer rescheduled to fire after 86400 secs
00000a4c.00001f94::2015/12/12-08:05:04.196 INFO  [GUM] Node 2: Processing RequestLock 1:1663
00000a4c.00001f94::2015/12/12-08:05:04.219 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 6244)
00000cb4.00001074::2015/12/12-09:00:58.315 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
00000a4c.00002a30::2015/12/12-09:00:58.315 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
00000cb4.00002324::2015/12/12-09:00:58.630 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
00000a4c.00002a30::2015/12/12-09:00:58.630 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
00000a4c.00001a9c::2015/12/12-09:56:16.160 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002b64::2015/12/12-09:56:18.136 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000028b8::2015/12/12-09:56:18.136 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:56:18.136 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001bb4::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a2c::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002b04::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000016ec::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001440::2015/12/12-09:56:19.484 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a90::2015/12/12-09:56:20.164 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:56:21.346 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000028b8::2015/12/12-09:56:21.839 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001a9c::2015/12/12-09:56:21.839 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002b64::2015/12/12-09:56:21.839 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a18::2015/12/12-09:56:21.839 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001440::2015/12/12-09:56:21.839 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:56:21.839 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002b64::2015/12/12-09:56:57.547 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a18::2015/12/12-09:57:01.310 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.310 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.726 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000022d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a90::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000016ec::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002b04::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a2c::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001bb4::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000002d8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000028b8::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001a9c::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000006fc::2015/12/12-09:57:01.727 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002c98::2015/12/12-09:57:01.833 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00000270::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.0000287c::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002a18::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00001a9c::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.0000186c::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.000028b8::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:57:03.831 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002408::2015/12/12-09:57:37.813 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.
00000a4c.00002b64::2015/12/12-09:57:44.918 INFO  [Timer] Ignoring callback because we don't allow concurrent callbacks.

Where can I get detailed info about message [RCM] rcm::RcmApi::FailResource: (AG) from first log and what does message [Timer] Ignoring callback because we don't allow concurrent callbacks. from second log mean?

UPDATE:

Node1

Microsoft SQL Server 2014 – 12.0.4100.1 (X64) Apr 20 2015 17:29:27
Copyright (c) Microsoft Corporation Enterprise Edition: Core-based
Licensing (64-bit) on Windows NT 6.1 (Build 7601: Service Pack
1)

Max degree of parallelism – 1

Total memory – 131037 MB

Max memory – 122880 MB

Node2

Microsoft SQL Server 2014 – 12.0.4100.1 (X64) Apr 20 2015 17:29:27
Copyright (c) Microsoft Corporation Enterprise Edition: Core-based
Licensing (64-bit) on Windows NT 6.1 (Build 7601: Service Pack
1)

Max degree of parallelism – 0

Total memory – 16383 MB

Max memory – 8192 MB

Best Answer

It looks like you are experiencing an error with your Windows clustering. A look at the error that you have on the 2nd node:

[RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
[RCM] Failed to load restype 'MSMQ': error 21.
[RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
[RCM] Failed to load restype 'MSMQTriggers': error 21.

This is explained in MSMQ Errors in the Cluster.log on the Failover Clustering and Network Load Balancing Team Blog, where it says:

Root Cause:

These events are logged because the MSMQ and MSMQ Triggers resource types are registered with the cluster service, but the MSMQ resource DLL cannot be loaded because the MSMQ feature is not installed. The MSMQ and MSMQ Triggers resource types are registered by default when the Failover Clustering feature is installed.

This would only be a problem if you are using MSMQ otherwise it can be ignored as benign.

Related Question