Linux – Poor IO due to LUKS/Software RAID/LVM ordering

encryptioniolinuxlukssoftware-raid

I'm trying to determine if I should re-setup my RAID array due to poor IO performance. First off, the system:

  • i7 920
  • 4 4TB WD 5400 Green drives
  • CentOS 6.3 host

Secondly, the disk setup:

  • /dev/sda2,b2,c2,d2 are individually LUKS encrypted
  • /dev/mapper/a2,b2,c2,d2 are all part of a software RAID5 /dev/md1
  • /dev/md1 has LVM on top of that
  • LVM is used to separate /, /storage, and swap

I choose this structure to allow for multiple instances of kcryptd, thinking that by doing this, I would get multithread support on encryption since one instance is running per drive. However, I'm beginning to wonder if that was a good idea.

For instance, if I run a heavy decompression routine on a RAR file of random data, my IO Wait goes up to around 25% and it slows the overall system down. I'm wondering if all the instruction sets are getting backed up somehow due to all the kcryptd processes.

Therefore, I'm considering changing to:

  • /dev/sda2,b2,c2,d2 are put into /dev/md1
  • /dev/md1 is encrypted and mapped to /dev/mapper/1
  • LVM on top of /dev/mapper/1

This would drop down to a single kcrpytd process, which could be a bottleneck in it's own right, too. Does anyone think this will help with my IO issue?

Best Answer

Your layering is suboptimal because putting the raid 5 on top of the encryption means that you increase the number of encrypt/decrypt operations by 25 % - since 4 * 4 TB are encrypted.

When putting the encryption on top of the raid 5 only 3 * 4 TB are encrypted.

The reasoning behind that is: you don't have to encrypt parity data (which takes up 4 TB in your example) of encrypted data because it does not increase your security.

Your presumption about multiple kcrypt processes is just that. When basing decisions on it, it is a premature optimization that may have quite the opposite effect. Your i7 is quite beefy, probably even including some special instructions that help to speed up AES - and the Linux kernel includes several optimized variants of cryptographic primitives that are automatically selected during boot.

You can verify if the optimized routines for your CPU are used via looking at /proc/cpuinfo (e.g. flag aes there), /proc/crypto, lsmod (unless the aes modules are compiled into the kernel) and the kernel log.

You should benchmark the throughput of kryptd without involving any slow disks to see what the upper bound really is (i.e. on a RAM disk using iozone).

To be able to diagnose potential performance issues later it is also useful to benchmark your RAID-setup of choice without any encryption to get an upper bound on that end.

In addition to the crypto topic, RAID 5 involves more IO-Operations than RAID 1 or 10. Since storage is kind of cheap perhaps it is an option to buy more harddisks and use another RAID level.

Related Question