Btrfs balance fails due to enospc error

btrfsopensuseraid

[taken over from stackoverflow since I figured that was probably the wrong place]

I am currently testing btrfs as a possible replacement for my NAS running ZFS. I created 4 devices, 20GB each, and created a btrfs filesystem using the balance -dconvert=raid5 option. Then I started to fill the FS using 5GB files (right now I have 9 of them). After creating these files (using dd from dev/zero) I started balance btrfs balance start. After some time it throws the error error during balancing '.' - No space left on device. The syslog brings up this error: btrfs: 4 enospc errors during balance.

Running balance again with -dusage=0 completes successfully (as expected) but balance -dusage=90 or simply balance without filters does not work. Why am I caring about this so much? I tested what happens when you remove a harddisk from a raid5 btrfs volume to see if no data corruption takes place. After removing the harddisk I wanted to add another and remove the failed one. This process failed due to enospc errors as well.

Additional information below:

(the mountpoint for the FS is /mnt/raid/)

linux:/mnt/raid # btrfs filesystem show
Label: 'test_raid5'  uuid: 8b137115-974e-4f91-a243-ec6239c34761
    Total devices 4 FS bytes used 45.06GiB
    devid    4 size 20.00GiB used 16.03GiB path /dev/sde
    devid    3 size 20.00GiB used 17.00GiB path /dev/sdd
    devid    2 size 20.00GiB used 17.00GiB path /dev/sdc
    devid    1 size 20.00GiB used 16.04GiB path /dev/sdb

Label: none  uuid: 69c33622-1c80-4dc3-9c01-24c27def730c
    Total devices 1 FS bytes used 1.04GiB
    devid    1 size 18.92GiB used 3.04GiB path /dev/sda2

linux:/mnt/raid # btrfs filesystem df /mnt/raid/
Data, RAID5: total=48.00GiB, used=45.01GiB
System, RAID1: total=32.00MiB, used=12.00KiB
System: total=4.00MiB, used=0.00
Metadata, RAID1: total=1.00GiB, used=53.69MiB


linux:/mnt/raid # btrfs version
Btrfs v0.20-rc1+20130701


linux:/mnt/raid # uname -a
Linux linux 3.11.6-4-default #1 SMP Wed Oct 30 18:04:56 UTC 2013 (e6d4a27) x86_64 x86_64 x86_64 GNU/Linux


linux:/mnt/raid # cat /etc/os-release
NAME=openSUSE
VERSION="13.1 (Bottle)"
VERSION_ID="13.1"
PRETTY_NAME="openSUSE 13.1 (Bottle) (x86_64)"
ID=opensuse
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:opensuse:13.1"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://opensuse.org/"
ID_LIKE="suse"

I cannot believe I have to add even more space to this just to be able to rebalance my btrfs volume. Any ideas on this?

Best Answer

I decided to take the road of btrfs instead of zfs for my "NAS". I tested both before, and I have to admit that I found ZFS much better documented, I was really seduced by its command line interface, so for me ZFS was the "better" choice.

But there was a show-stopper: if you have 3x 2TB HDD for ZFS in RAID-5 and you want to grow them to 4x TB HDD, that is not easy. If you ZFS pool as 3HDD and you want to add new HDD, then you need to add 3 more: for a total of 6x 2TB. And if you have 6HDD and want to grow your FS, you need to add 6 more, etc. I had the problem that my NAS enclosure can't hold more than 4 disks and that I can't afford buying yet the 4 HDDs. In this respect Btrfs is much more flexible.

However, RAID-5/6 was really badly supported when I started. I did some tests like you, and I ended up corrupting the filesystem. I did those same tests in RAID-1 and no problems! So I decided to go Btrfs in RAID-1, and instead of using the Ubuntu LTS, I'm always using the latest Ubuntu release to get the latest Kernel and btrfs-tools package (I'm using Ubuntu 15.04 and getting ready to switch to 15.10). I will probably (after testing) finally move to RAID-5 once Ubuntu 16.04 is released and probably stick to it. I've seen in recent kernel changelogs that the code base for RAID-5/6 is much more mature.

So to your problem. If you want to try Btrfs especially in RAID-5 or 6, stick with the most up-to-date Linux distributions (e.g. Arch Linux, latest Ubuntu, Fedora, etc.). They might lack the stability of other distributions, but for Btrfs they are probably much more stable.

One note about your "use case": your Btrfs filesystem is rather small compared to the size of the files you put in. It probably doesn't really well reflect a real world scenario. You should create big (videos) and small files (photos, docs) and try to fill up your file system up to 80%. One good way to test it is to put many files on your file system and then run filesystem benchmarks, not for the benchmarking itself but for the stress they bring. You can use the phoronix-test-suite for this (e.g. phoronix-test-suite benchmark pts/disk).

Related Question