Choosing filesystem for a 16TB Raid

filesystemslvmraid

I'm just putting together a machine with eight 2TB disks.

I will be using Raid 6 (12TB of usable capacity) on top of them, but I'm not sure whether I should make LVM on top of the RAID, or what filesystem to use.

What filesystems can be resized when used inside LVM?

Best Answer

With lvm on top of a raid device you are flexible to create multiple virtual devices (and filesystems) on it. And you are flexible to change the size of those devices.

If you are 100% sure that you don't need that and you only need one big filesystem, then you can directly create the filesystem on your raid device. One layer of indirection and complexity is removed in that case.

To choose a filesystem, the most important points are:

  • should be well tested and stable
  • should be mainstream enough
  • good performance of course

That means one is usually conservative when it comes to filesystems.

Using these criteria you have basically 3 choices on Linux (as of 2011-06:

  • ext3
  • ext4
  • xfs

On big devices I use xfs because a mkfs.xfs is way faster.

All of these filesystems can be resized.

Update:

I did a small benchmark on a 3 TB device (using 4k blocksize in all filesystems):

$ awk -F\; -f mkfs.awk mkfs
          FS     SIZE(TB)      TIME(S)      RSS(MB)      SPEEDUP      SPACEUP
        ext3            1          217           37         1.00         1.00
        ext3            2          478           74         1.00         1.00
        ext3            3          829          111         1.00         1.00
        ext4            1          139           37         1.55         1.00
        ext4            2          298           74         1.60         1.00
        ext4            3          515          111         1.61         1.00
         xfs            1            5            2        43.23        17.01
         xfs            2            9            2        51.43        33.49
         xfs            3           15            2        54.73        50.05

(The speed/mem-up is against ext3)

(System: Debian 6.0 amd64, mkfs.ext 1.41.12, mkfs.xfs 3.1.4, WD SATA drive, hdparm -t about 120 MB/s buffered disk reads)

That means mkfsing a ext[34] filesystem is up to 54 times slower than mkfsing a xfs one. Approximating this to a 12 TB creating a ext fs would really take about an hour (xfs only about a minute).

Related Question