extended-eventssql serversql-server-2017
Does anyone know how to tell how many resources (CPU/MEM/Reads etc) have been used by SQL Server Extended Events?
I've went through the available DMVs here
But I dont see where this is tracked anywhere.
I'm asking b/c I worked on a poorly performing sql server for a few days before I realized there was an EE session running that, when killed, cut CPU utilization in half. (It was Idera Sql DM running the EE Session)
The reason for the performance difference lies in how scalar expressions are handled in the execution engine. In this case, the expression of interest is:
[Expr1000] = CONVERT(xml,DM_XE_SESSION_TARGETS.[target_data],0)
This expression label is defined by a Compute Scalar operator (node 11 in the serial plan, node 13 in the parallel plan). Compute Scalar operators are different from other operators (SQL Server 2005 onward) in that the expression(s) they define are not necessarily evaluated at the position they appear in the visible execution plan; evaluation can be deferred until the result of the computation is required by a later operator.
In the present query, the target_data string is typically large, making the conversion from string to XML expensive. In slow plans, the string to XML conversion is performed every time a later operator that requires the result of Expr1000 is rebound.
Rebinding occurs on the inner side of a nested loops join when a correlated parameter (outer reference) changes. Expr1000 is an outer reference for most of the nested loops joins in this execution plan. The expression is referenced multiple times by several XML Readers, both Stream Aggregates, and by a start-up Filter. Depending on the size of the XML, the number of times the string is converted to XML can easily number in the millions.
The call stacks below show examples of the target_data string being converted to XML (ConvertStringToXMLForES - where ES is the Expression Service):
Start-up Filter
XML Reader (TVF Stream internally)
Stream Aggregate
Converting the string to XML each time any of these operators rebind explains the performance difference observed with the nested loops plans. This is regardless of whether parallelism is used or not. It just so happens that the optimizer chooses a hash join when the MAXDOP 1 hint is specified. If MAXDOP 1, LOOP JOIN is specified, performance is poor just as with the default parallel plan (where the optimizer chooses nested loops).
How much performance increases with a hash join depends on whether Expr1000 appears on the build or probe side of the operator. The following query locates the expression on the probe side:
SELECT CAST (
REPLACE (
REPLACE (
XEventData.XEvent.value ('(data/value)[1]', 'varchar(max)'),
'<victim-list>', '<deadlock><victim-list>'),
'<process-list>', '</victim-list><process-list>')
AS XML) AS DeadlockGraph
FROM (SELECT CAST (target_data AS XML) AS TargetData
FROM sys.dm_xe_sessions s
INNER HASH JOIN sys.dm_xe_session_targets st ON s.address = st.event_session_address
WHERE [name] = 'system_health') AS Data
CROSS APPLY TargetData.nodes ('//RingBufferTarget/event') AS XEventData (XEvent)
WHERE XEventData.XEvent.value('@name', 'varchar(4000)') = 'xml_deadlock_report';
I have reversed the written order of the joins from the version shown in the question, because join hints (INNER HASH JOIN above) also force the order for the whole query, just as if FORCE ORDER had been specified. The reversal is necessary to ensure Expr1000 appears on the probe side. The interesting part of the execution plan is:
With the expression defined on the probe side, the value is cached:
Evaluation of Expr1000 is still deferred until the first operator needs the value (the start-up filter in the stack trace above) but the computed value is cached (CValHashCachedSwitch) and reused for later calls by the XML Readers and Stream Aggregates. The stack trace below shows an example of the cached value being reused by an XML Reader.
When the join order is forced such that the definition of Expr1000 occurs on the build side of the hash join, the situation is different:
SELECT CAST (
REPLACE (
REPLACE (
XEventData.XEvent.value ('(data/value)[1]', 'varchar(max)'),
'<victim-list>', '<deadlock><victim-list>'),
'<process-list>', '</victim-list><process-list>')
AS XML) AS DeadlockGraph
FROM (SELECT CAST (target_data AS XML) AS TargetData
FROM sys.dm_xe_session_targets st
INNER HASH JOIN sys.dm_xe_sessions s ON s.address = st.event_session_address
WHERE [name] = 'system_health') AS Data
CROSS APPLY TargetData.nodes ('//RingBufferTarget/event') AS XEventData (XEvent)
WHERE XEventData.XEvent.value('@name', 'varchar(4000)') = 'xml_deadlock_report'
A hash join reads its build input completely to construct a hash table before it starts probing for matches. As a result, we have to store all the values, not just the one per thread being worked on from the probe side of the plan. The hash join therefore uses a tempdb work table to store the XML data, and every access to the result of Expr1000 by later operators requires an expensive trip to tempdb:
The following shows more details of the slow access path:
If a merge join is forced the input rows are sorted (a blocking operation, just like the build input to a hash join) resulting in a similar arrangement where slow access via a tempdb sort-optimized worktable is required because of the size of the data.
Plans that manipulate large data items can be problematic for all sorts of reasons that are not apparent from the execution plan. Using a hash join (with the expression on the correct input) is not a good solution. It relies on undocumented internal behaviour with no guarantees it will work the same way next week, or on a slightly different query.
The message is that XML manipulation can be tricky things to optimize today. Writing the XML to a variable or temporary table before shredding is a much more solid workaround than anything shown above. One way to do this is:
DECLARE @data xml =
CONVERT
(
xml,
(
SELECT TOP (1)
dxst.target_data
FROM sys.dm_xe_sessions AS dxs
JOIN sys.dm_xe_session_targets AS dxst ON
dxst.event_session_address = dxs.[address]
WHERE
dxs.name = N'system_health'
AND dxst.target_name = N'ring_buffer'
)
)
SELECT XEventData.XEvent.value('(data/value)[1]', 'varchar(max)')
FROM @data.nodes ('./RingBufferTarget/event[@name eq "xml_deadlock_report"]') AS XEventData (XEvent)
WHERE XEventData.XEvent.value('@name', 'varchar(4000)') = 'xml_deadlock_report';
Finally, I just want to add Martin's very nice graphic from the comments below:
I know this question, based on the Title, is mainly concerned with the PREEMPTIVE_OS_DELETESECURITYCONTEXT wait type, but I believe that is a misdirection of the true issue which is " a customer who was complaining about high CPU usage on their SQL Server ".
The reason I believe that focusing on this specific wait type is a wild goose chase is because it goes up for every connection made. I am running the following query on my laptop (meaning I am the only user):
SELECT *
FROM sys.dm_os_wait_stats
WHERE wait_type = N'PREEMPTIVE_OS_DELETESECURITYCONTEXT'
And then I do any of the following and re-run this query:
SQLCMD -E -Q "select 1"Now, we know that CPU is high so we should look at what is running to see what sessions have high CPU:
SELECT req.session_id AS [SPID],
req.blocking_session_id AS [BlockedBy],
req.logical_reads AS [LogReads],
DB_NAME(req.database_id) AS [DatabaseName],
SUBSTRING(txt.[text],
(req.statement_start_offset / 2) + 1,
CASE
WHEN req.statement_end_offset > 0
THEN (req.statement_end_offset - req.statement_start_offset) / 2
ELSE LEN(txt.[text])
END
) AS [CurrentStatement],
txt.[text] AS [CurrentBatch],
CONVERT(XML, qplan.query_plan) AS [StatementQueryPlan],
OBJECT_NAME(qplan.objectid, qplan.[dbid]) AS [ObjectName],
sess.[program_name],
sess.[host_name],
sess.nt_user_name,
sess.total_scheduled_time,
sess.memory_usage,
req.*
FROM sys.dm_exec_requests req
INNER JOIN sys.dm_exec_sessions sess
ON sess.session_id = req.session_id
CROSS APPLY sys.dm_exec_sql_text(req.[sql_handle]) txt
OUTER APPLY sys.dm_exec_text_query_plan(req.plan_handle,
req.statement_start_offset,
req.statement_end_offset) qplan
WHERE req.session_id <> @@SPID
ORDER BY req.logical_reads DESC, req.cpu_time DESC
--ORDER BY req.cpu_time DESC, req.logical_reads DESC
I usually run the above query as it is, but you could also switch which ORDER BY clause is commented out to see if that gives more interesting / helpful results.
Alternatively you can run the following, based on dm_exec_query_stats, to find highest-cost queries. The first query below will show you individual queries (even if they have multiple plans) and is ordered by Average CPU Time, but you can easily change that to be Average Logical Reads. Once you find a query that looks like it is taking a lot of resources, copy the "sql_handle" and "statement_start_offset" into the WHERE condition of the second query below to see the individual plans (can be more than 1). Scroll to the far right and assuming there was an XML Plan, it will display as a link (in Grid Mode) which will take you to the plan viewer if you click on it.
Query #1: Get Query Info
;WITH cte AS
(
SELECT qstat.[sql_handle],
qstat.statement_start_offset,
qstat.statement_end_offset,
COUNT(*) AS [NumberOfPlans],
SUM(qstat.execution_count) AS [TotalExecutions],
SUM(qstat.total_worker_time) AS [TotalCPU],
(SUM(qstat.total_worker_time * 1.0) / SUM(qstat.execution_count)) AS [AvgCPUtime],
MAX(qstat.max_worker_time) AS [MaxCPU],
SUM(qstat.total_logical_reads) AS [TotalLogicalReads],
(SUM(qstat.total_logical_reads * 1.0) / SUM(qstat.execution_count)) AS [AvgLogicalReads],
MAX(qstat.max_logical_reads) AS [MaxLogicalReads],
SUM(qstat.total_rows) AS [TotalRows],
(SUM(qstat.total_rows * 1.0) / SUM(qstat.execution_count)) AS [AvgRows],
MAX(qstat.max_rows) AS [MaxRows]
FROM sys.dm_exec_query_stats qstat
GROUP BY qstat.[sql_handle], qstat.statement_start_offset, qstat.statement_end_offset
)
SELECT cte.*,
DB_NAME(txt.[dbid]) AS [DatabaseName],
SUBSTRING(txt.[text],
(cte.statement_start_offset / 2) + 1,
CASE
WHEN cte.statement_end_offset > 0
THEN (cte.statement_end_offset - cte.statement_start_offset) / 2
ELSE LEN(txt.[text])
END
) AS [CurrentStatement],
txt.[text] AS [CurrentBatch]
FROM cte
CROSS APPLY sys.dm_exec_sql_text(cte.[sql_handle]) txt
ORDER BY cte.AvgCPUtime DESC
Query #2: Get Plan Info
SELECT *,
DB_NAME(qplan.[dbid]) AS [DatabaseName],
CONVERT(XML, qplan.query_plan) AS [StatementQueryPlan],
SUBSTRING(txt.[text],
(qstat.statement_start_offset / 2) + 1,
CASE
WHEN qstat.statement_end_offset > 0
THEN (qstat.statement_end_offset - qstat.statement_start_offset) / 2
ELSE LEN(txt.[text])
END
) AS [CurrentStatement],
txt.[text] AS [CurrentBatch]
FROM sys.dm_exec_query_stats qstat
CROSS APPLY sys.dm_exec_sql_text(qstat.[sql_handle]) txt
OUTER APPLY sys.dm_exec_text_query_plan(qstat.plan_handle,
qstat.statement_start_offset,
qstat.statement_end_offset) qplan
-- paste info from Query #1 below
WHERE qstat.[sql_handle] = 0x020000001C70C614D261C85875D4EF3C90BD18D02D62453800....
AND qstat.statement_start_offset = 164
-- paste info from Query #1 above
ORDER BY qstat.total_worker_time DESC
Best Answer
It should be more of a case support with idera. Generally these are good repuatated products in market which have a better documentation than we can explain here. I can share this based on my experience with sentry support where had similar if not exact issue. E.g if you are using idera to collect actual execution plan or events for tracking queries and their metrics comes with additional cost.
Please open a case with idera and go through their documentation for further troubleshooting.