We started getting Alerts:
Could not connect because the maximum number of '1' dedicated administrator connections already exists. Before a new connection can be made, the existing dedicated administrator connection must be dropped, either by logging off or ending the process. [CLIENT: 127.0.0.1]
With a help of SQL job and holding table, we managed to find out that it's the SQL Server Service Account that tries to connect, approx every 4 hours e.g. 12:00 16:00 and so on, with a program name: Net SqlClient Data Provider.
I've added the host_process_id column as per jadarnel27 suggestions, identified the PID, but there were no processes with the same PID in the task manager.
Oddly enough, there are no entries in the log: "Dedicated admin connection support was established", just the following ones:
16:00:05 Login succeeded for user 'sql service user'. Connection made using Windows authentication. [CLIENT: ip address of Windows Failover Cluster Virtual Adapter]
16:00:05 Could not connect because the maximum number of '1' dedicated administrator connections already exists. Before a new connection can be made, the existing dedicated administrator connection must be dropped, either by logging off or ending the process. [CLIENT: 127.0.0.1]
16:00:11 Dedicated administrator connection has been disconnected. This is an informational message only. No user action is required.
No clues in Windows Event Viewer logs or Windows Cluster logs.
This is only happening on an active node of AlwaysOn Availability Group.
No jobs/port scanning configured to run at those times, nor any other tasks.
I have seen How do we find who used Dedicated Admin Connection but we know what account has used DAC, just don't understand why.
How can we troubleshoot the issue further?
EDIT: I've found the cause of the problem, it was a powershell script that synchronises server objects for Availability Groups, it was set to run on port 1433 and I have no idea why it tried to connect to DAC, however simple reboot has solved the issue! Thanks ever so much to everyone who replied!
Best Answer
I recon that this is port/virus scanner that your security team might be using to find out vulnerabilities e.g. weak passwords. I had the same issue and eventually I figured out that it was the port scanner.
Look into windows event log as well.