Sql-server – Physical only checkdb is failing, but full one is completed successfully

dbcc-checkdbsql serversql-server-2017

I am executing checkdb with physical_only option and it fails with multiple errors like below:

Msg 8965, Level 16, State 1, Line 1
Table error: Object ID 1557580587,
index ID 1, partition ID 72057594088456192, alloc unit ID
72057594177454080 (type In-row data). The off-row data node at page
(1:13282192), slot 3, text ID 6370769698816 is referenced by page
(0:0), slot 0, but was not seen in the scan.
Msg 8965, Level 16, State
1, Line 1
Table error: Object ID 1557580587, index ID 1, partition ID
72057594088456192, alloc unit ID 72057594177454080 (type In-row data).
The off-row data node at page (1:13282192), slot 5, text ID
6370769764352 is referenced by page (0:0), slot 0, but was not seen in
the scan.
CHECKDB found 0 allocation errors and 5255 consistency
errors in table 'TableX' (object ID
1557580587).
CHECKDB found 0 allocation errors and 5255 consistency
errors in database 'DatabaseX'. repair_allow_data_loss is the minimum
repair level for the errors found by DBCC CHECKDB (DWH_LAND).

However full checkdb is successfull:

CHECKDB found 0 allocation errors and 0 consistency errors in database
'DatabaseX'. DBCC execution completed. If DBCC printed error messages,
contact your system administrator.

TableX has around 200 000 rows and has clustered columnstore index on it.

We are using following version of SQL Server:

Microsoft SQL Server 2017 (RTM-CU13) (KB4466404) – 14.0.3048.4

Should I be worried?

Best Answer

Yes, this a defect in SQL Server 2017 up to CU14. There was a similar issue with nonclustered columnstore indexes in SQL Server 2016.

It's detailed in this video: CHECKDB Bug With SQL Server 2017 And ColumnStore Indexes

A more portable repro of the issue is this:

SELECT *, CONVERT(NVARCHAR(MAX), m.text + m.text + m.text + m.text + m.text) AS texty
INTO dbo.corrupt
FROM sys.messages AS m;

CREATE CLUSTERED COLUMNSTORE INDEX cx_corrupt ON dbo.corrupt;

DBCC CHECKDB WITH PHYSICAL_ONLY;

DROP TABLE dbo.corrupt;

Related Solutions

Sql-server – DBCC CHECKDB Results: page ID contains an incorrect page ID in its page header

Never use REPAIR_ALLOW_DATA_LOSS. I would do a database restore before that.

I suggest following this order:

REPAIR_FAST (works 1 of 10 times).
REPAIR_REBUILD (works up to 50% of the time).
Restore database.

Full syntax:

DBCC CHECKALLOC   
[  
    ( database_name | database_id | 0   
      [ , NOINDEX   
      | , { REPAIR_ALLOW_DATA_LOSS | REPAIR_FAST | REPAIR_REBUILD } ]  
    )  
    [ WITH   
        {   
          [ ALL_ERRORMSGS ]  
          [ , NO_INFOMSGS ]   
          [ , TABLOCK ]   
          [ , ESTIMATEONLY ]   
        }  
    ]  
]

Sql-server – Corrupt DB for a noob

It looks as though you have a problem with your storage subsystem (somewhere from the drivers down to the actual disks, but it could be anywhere in that stack).

The good news:

Most of the corruption is involved in non-clustered indexes. This means that if the underlying table is clean, the indexes can be rebuilt to fix the corruption. Object IDs 1345738380, 1761739862 (this has clustered index corruption as well), 2056692908
It doesn't seem like any PFS/GAM/SGAM/etc system pages were damaged.

The bad news:

There are some pages corrupt in the clustered index. This means there could (most likely will) be data loss without a recent backup. Object ID 1761739862
You don't have a recent backup.
You already ran checkdb with repair_allow_data_loss so there is no way to know what you've already lost or what has already been done which takes you an extra step back.
It looks like there is an issue with the disk subsystem.

Where to go from here:

I'd start by taking a backup with continue_after_error to make sure that you have something of a record. Then restore that backup to an instance on the same patch level as the one it was taken from so that all testing can be completed on a copy of the database and not the database itself.

Use the copy and the older backup of the database you have on a second instance to see what data is lost and what may be able to be manually salvageable. This is time consuming and lengthy, but may be needed. If you're really stuck, call in a consultant to help you with this as corruption is a great thing to cut your teeth on and gives you some semblance of liability.

Check the objects associated with the corruption. If they are tables that don't have any super important data (say dictionary tables that can be rebuilt or don't change much) you might be able to get away with manually fixing the table by using the older backup to script out data.

Check your storage subsystem, make sure you have the latest drivers, firmware, etc. Check for any failed drives in the array/san/nas/etc. Double check Ethernet cables, fibre cables, switches, etc. Find the root cause of the corruption or this may happen again. Run a health check on your storage subsystem/motherboard to make sure something isn't faulty with the hardware or controllers.

Lastly, update your resume.

Best Answer

Related Solutions

Sql-server – DBCC CHECKDB Results: page ID contains an incorrect page ID in its page header

Sql-server – Corrupt DB for a noob

Related Question