Linux Kernel – Defining State of Failed SD Cards by Kernel Tracing

This seems more an hardware/use case problem than something else. I bet common sense might be more important than Linux skills here. Are you doing heavy I/O in the cards, MySQL/Apache/compiling stuff...syslog/frequent system updates? -- comment by Rui F Ribeiro

I can expand on the above. But I agree with the first point, and I agree this was the first question to ask.

• Should I enable some debug logs for the MMC subsystem?
• Is there a userspace tool that can sniff what's going on?
• How do I make the error codes make more sense?

The only confidence I've had from attributing failures has come more from the "history" and general results I get, not the specific errors from low-level commands. Which are likely to vary between implementations anyway.

Even with an SSD, from a reasonable brand, I believe I've had bad data returned in place of I/O errors. This has certainly been one of the known failure modes across many SSDs. [2013][2017]. (Possibly surprising to people familiar with the contemporary filesystems and database implementations which often hope for a more manageable set of failure modes). Notice that the papers I link here focused on the data returned; they did not make any more distinction in reported errors, except for the dead drive / bad sector distinction which you have already measured.

My SSD fault was on a "seller refurbished" laptop, which had already been "repaired" once, and was starting to show failures again - plausibly causing a power interruption to the drive just as in the linked papers. It might also have failed to provide stable voltage levels.

I'm trying to understand if it's a matter of bad SD cards or if there might be something wrong with the controller driver that is pushing the cards to corruption.

Good hardware with a good mains power supply doesn't tend to destroy a good SD card - unless you're putting too much load on it. The workload is a very important variable, which you did not [originally] mention. These memory cards are relatively small, usually cheap hardware that is designed for relatively un-demanding use storing media files (hence MMC, "MultiMediaCard"). Particularly cheaper ones won't necessarily be very good at "wear-levelling" (redistributing the load from hotspot logical blocks across a large number of physical blocks).

I have measured workload with a quick hack, scheduling a daily cron job to run tunefs -l /dev/mmcblk0p4 | grep writes >> /var/log/writes.log.

But if we set the workload aside, you'd be right to consider a possible controller-side issue from the information you gave so far. I've had repeated bad sectors on an SD card due to writes from a pocket device, possibly when its battery was low. This was a card from the one name brand. The sectors were recoverable and I'm still using the same card. I've also had some sort of transient initialization failure on this card, I think it was associated with bad sectors too (once I got past the initialization failure), but I could be mis-remembering.

I'm having a series of failing/sometimes failing SD cards.

The impression I get from your [original] question is that this is a small scale operation, and running a rigorous test matrix with different cards, controllers, and workloads would be overkill.

After the workload, the first variable you control is the card.

Writing in 2018, there is one global name brand which can be considered "canonical" for sd cards -

see results at: https://www.amazon.com/s/field-keywords=sd+card

- and you hopefully have a number of retail channels that can be considered... at least reliable enough for comparison purposes. (Remembering that various popular online retailers act as "marketplace" as well as selling their own goods).

Official Raspbery PI hardware might also be acceptible. I.e. SD cards, sold officially for running Linux on a small board computer, which have been reported to work well. (Being a more demanding workload than media files).

As a broad brush, if you get a card that's faster than you strictly need, I also think of that as a potentially higher endurance rating. (Given that speed rating tends to be much more available than endurance).

If you control / measure these two variables, then you can focus your judgements on the rest of the relevant hardware.

The failing ones (can occasionally still be mounted)

Note, in the most general case, if you think a device has been badly written, you may attempt to clear this fault:

1. recover what data you can if desired
2. then stop trying to read bad blocks. simply recreate the entire formatting (partition table + filesystem).
3. but if you're not sure and think the device might still be dying, you probably also want to test it.

If you have nice native MMC hardware like you do, you can use the Linux command blkdiscard as a more efficient way to test erasing all the blocks of the device, before you "reformat" it. But efficiency is the only advantage, compared to testing for errors when overwriting the whole drive with zeros i.e. dd bs=1M if=/dev/zero of=/dev/mmcblk0. (As well as avoiding any need to write the erased blocks, blkdiscard could also in theory provide a more "as-new" performance afterwards, and increase endurance, by giving the device a bit more freedom).

(If this was a SATA drive - there is a dedicated "secure erase" command to discard the entire logical drive contents (see man hdparm). However I am not aware of any equivalent MMC command. Certain SSD vendors took advantage of this command to reset their block mapping tables, as a workaround for their failure to recover "as-new" performance with the equivalent blkdiscard sequence. Note this command does not necessarily test a full-drive erase. In some cases it will only erase an internal encryption key).

Since you asked what my errors looked like

My SanDisk micro-SD card played up again recently. It seems the specific errors below were due to flaky connection. It was resolved by removing & re-inserting the micro-SD into the micro-SD to SD adaptor, after superstitiously blowing on all the metal pads.

In the reader on my Dell Latitude E5450 laptop (sdhci-pci kernel driver, Fedora Linux kernel version circa v4.17), it was failing to initialize the card. On my SheevaPlug (same hardware and software details as this question), this card seems to have been able to initialize but it showed IO errors. Perhaps on the Dell the error-handling timeouts are not quite set up correctly.

Dell:

[    2.436566] mmc0: Unknown controller version (3). You may experience problems.
[    2.449019] mmc0: SDHCI controller on PCI [0000:01:00.0] using ADMA
...
[509227.374012] mmc0: error -84 whilst initialising SD card
[509227.621510] mmc0: error -84 whilst initialising SD card
[509227.865472] mmc0: error -84 whilst initialising SD card
[509228.142120] mmc0: error -84 whilst initialising SD card

Sheevaplug:

[6076613.118617] mmcblk0: mmc0:aaaa SC16G 14.8 GiB 
[6076613.295811] mmcblk0: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
[6076613.545740] mmcblk0: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
[6076613.555301] mmcblk0: retrying using single block read
[6076613.728413] mmcblk0: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
[6076613.737965] blk_update_request: I/O error, dev mmcblk0, sector 0
[6076613.912043] mmcblk0: error -110 transferring data, sector 1, nr 7, cmd response 0x900, card status 0x0
[6076613.921599] blk_update_request: I/O error, dev mmcblk0, sector 1
...