I have a persisted computed column on a table which is simply made up concatenated columns, e.g.
CREATE TABLE dbo.T
(
ID INT IDENTITY(1, 1) NOT NULL CONSTRAINT PK_T_ID PRIMARY KEY,
A VARCHAR(20) NOT NULL,
B VARCHAR(20) NOT NULL,
C VARCHAR(20) NOT NULL,
D DATE NULL,
E VARCHAR(20) NULL,
Comp AS A + '-' + B + '-' + C PERSISTED NOT NULL
);
In this Comp
is not unique, and D is the valid from date of each combination of A, B, C
, therefore I use the following query to get the end date for each A, B, C
(basically the next start date for the same value of Comp):
SELECT t1.ID,
t1.Comp,
t1.D,
D2 = ( SELECT TOP 1 t2.D
FROM dbo.T t2
WHERE t2.Comp = t1.Comp
AND t2.D > t1.D
ORDER BY t2.D
)
FROM dbo.T t1
WHERE t1.D IS NOT NULL -- DON'T CARE ABOUT INACTIVE RECORDS
ORDER BY t1.Comp;
I then added an index to the computed column to assist in this query (and also others):
CREATE NONCLUSTERED INDEX IX_T_Comp_D ON dbo.T (Comp, D) WHERE D IS NOT NULL;
The query plan however surprised me. I would have thought that since I have a where clause stating that D IS NOT NULL
and I am sorting by Comp
, and not referencing any column outside of the index that the index on the computed column could be used to scan t1 and t2, but I saw a clustered index scan.
So I forced the use of this index to see if it yielded a better plan:
SELECT t1.ID,
t1.Comp,
t1.D,
D2 = ( SELECT TOP 1 t2.D
FROM dbo.T t2
WHERE t2.Comp = t1.Comp
AND t2.D > t1.D
ORDER BY t2.D
)
FROM dbo.T t1 WITH (INDEX (IX_T_Comp_D))
WHERE t1.D IS NOT NULL
ORDER BY t1.Comp;
Which gave this plan
This shows that a Key lookup is being used, the details of which are:
Now, according to the SQL-Server documentation:
You can create an index on a computed column that is defined with a deterministic, but imprecise, expression if the column is marked PERSISTED in the CREATE TABLE or ALTER TABLE statement. This means that the Database Engine stores the computed values in the table, and updates them when any other columns on which the computed column depends are updated. The Database Engine uses these persisted values when it creates an index on the column, and when the index is referenced in a query. This option enables you to create an index on a computed column when Database Engine cannot prove with accuracy whether a function that returns computed column expressions, particularly a CLR function that is created in the .NET Framework, is both deterministic and precise.
So if, as the docs say "the Database Engine stores the computed values in the table", and the value is also being stored in my index, why is a Key Lookup required to get A, B and C when they are not referenced in the query at all? I assume they are being used to calculate Comp, but why? Also, why can the query use the index on t2
, but not on t1
?
N.B. I have tagged SQL Server 2008 because this is the version that my main problem is on, but I also get the same behaviour in 2012.
Best Answer
Columns
A, B, and C
are referenced in the query plan - they are used by the seek onT2
.The optimizer decided that scanning the clustered index was cheaper than scanning the filtered nonclustered index and then performing a lookup to retrieve the values for columns A, B, and C.
Explanation
The real question is why the optimizer felt the need to retrieve A, B, and C for the index seek at all. We would expect it to read the
Comp
column using a nonclustered index scan, and then perform a seek on the same index (alias T2) to locate the Top 1 record.The query optimizer expands computed column references before optimization begins, to give it a chance to assess the costs of various query plans. For some queries, expanding the definition of a computed column allows the optimizer to find more efficient plans.
When the optimizer encounters a correlated subquery, it attempts to 'unroll it' to a form it finds easier to reason about. If it cannot find a more effective simplification, it resorts to rewriting the correlated subquery as an apply (a correlated join):
It just so happens that this apply unrolling puts the logical query tree into a form that does not work well with project normalization (a later stage that looks to match general expressions to computed columns, among other things).
In your case, the way the query is written interacts with internal details of the optimizer such that the expanded expression definition is not matched back to the computed column, and you end up with a seek that references columns
A, B, and C
instead of the computed column,Comp
. This is the root cause.Workaround
One idea to workaround this side-effect is to write the query as an apply manually:
Unfortunately, this query will not use the filtered index as we would hope either. The inequality test on column
D
inside the apply rejectsNULLs
, so the apparently redundant predicateWHERE T1.D IS NOT NULL
is optimized away.Without that explicit predicate, the filtered index matching logic decides it cannot use the filtered index. There are a number of ways to work around this second side-effect, but the easiest is probably to change the cross apply to an outer apply (mirroring the logic of the rewrite the optimizer performed earlier on the correlated subquery):
Now the optimizer does not need to use the apply rewrite itself (so the computed column matching works as expected) and the predicate is not optimized away either, so the filtered index can be used for both data access operations, and the seek uses the
Comp
column on both sides:This would generally be preferred over adding A, B, and C as
INCLUDEd
columns in the filtered index, because it addresses the root cause of the problem, and does not require widening the index unnecessarily.Persisted computed columns
As a side note, it is not necessary to mark the computed column as
PERSISTED
, if you don't mind repeating its definition in aCHECK
constraint:The computed column is only required to be
PERSISTED
in this case if you want to use aNOT NULL
constraint or to reference theComp
column directly (instead of repeating its definition) in aCHECK
constraint.