Linux – bcache on md or md on bcache

bcacheblock-devicelinuxssd

bcache allows one or more fast disk drives such as flash-based solid state drives (SSDs) to act as a cache for one or more slower hard disk drives.

If I understand correctly,

  • an SSD* could be assigned to cache multiple backing HDDs, and then the resulting cached devices could be RAIDed with mdadm
    or
  • multiple HDDs could be RAIDed into a single backing md device and the SSD assigned to cache that

I'm wondering which is the saner approach. It occurs to me that growing a RAID5/6 may be simpler with one or other technique, but I'm not sure which!

Are there good reasons (eg growing the backing storage or anything else) for choosing one approach over the other (for a large non-root filesystem containing VM backing files)?

* by "an SSD" I mean some sort of redundant SSD device, eg a RAID1 of two physical SSDs

Best Answer

I think caching the whole md device make most sense.

Putting bcache to cache the whole md device sacrifices the whole idea of having raid, because it introduces another single point of failure.

  • OTH failurs of SSD disks are relatively rare, and bcache can be put into the writethrough/writearound mode (in contrast to the writeback mode), where there is no data stored only to the cache device, and failure of the cache doesn't kill the information in the raid makes it a relatively safe option.

  • Other fact is that there is significant computational overhead of soft RAID-5; when caching each spinning raid member separately, computer still has to re-calculate all the parities, even on cache hits

  • Obviously, you'd sacrifice some expensive ssd space, if you cache each spinning drive separately. - Unless you plan to use raided ssd cache.

  • Both options relatively don't affect the time of growing process - although the option with spinning drives being cached separately has potential to be slower due to more bus traffic.

It is fast and relatively simple process to configure bcache to remove the ssd drive, when you need to replace it. Thanks to the blocks it should be possible to migrate the raid setup both ways on-place.

You should also remember, that at the moment most (all?) live-CD distributions don't support bcache, so you can't simply access your data with such tools regardless of the bcache-mdraid layout option you chose.

Related Question