Ubuntu – BTRFS inside a KVM-VM on a qcow2 formatted image

btrfscopy on writekvmqemuUbuntu

I have a Ubuntu 14.04 application appliance that makes heavy use of BTRFS snapshots. The application is meant to be hypervisor agnostic and the snapshots need to be stored with the virtual machine image in case we need to troubleshoot an issue. Using the hypervisors built in snapshotting methods instead of BTRFS snapshotting would be more work than it's worth since it would require API access to the hypervisor from the VM which we don't want for security reasons. I can also remotely access the BTRFS snapshot filesystem subvolumes directly from the appliances command line via ssh without having to power down the machine or access a hypervisor API.

In the past I've only had to deal with deploying this application to vmware based hypervisors. I always used thin provisioning on my vmware disks and never noticed any performance issues. I use thin provisioning because I do a large amount of testing on this appliance and I tend to deploy many appliances at a time to run different tests in parallel.

I'm very careful about not over committing the storage by using scripts that ensure the disk growth doesn't run out of control. The I/O dip that happens when the thin provisioned disk needs to grow isn't all that noticeable either.

Now I need to support KVM and would very much like to keep Thin/sparse provisioning my disks, however I've read a few things that state mixing a CoW filesystem with another CoW filesystem is a bad idea due to over-redundant writes and disk fragmentation among other things. The common example given was running a VM with a qcow2 formatted disk stored on a BTRFS formatted volume. My situation would be the opposite. I want to have a BTRFS formatted filesystem in a VM running from a qcow2 image. I haven't found much on the particulars of the performance impact of BTRFS snapshots on top of a qcow2 image.

Question: Are there any other sparse file formats that grow with the disk size that KVM likes to use?

I've explored with using sparse raw files, but I can't seem to get them to stay sparse through a cp, download untar/gunzip, ect. It seems that if you want to use a sparse raw file, you can't move it around at all which would make distribution a pain in the ass.

Best Answer

The thing is, KVM doesn't support disk images at all. Unlike vmware which is an all-in-one type of solution, KVM only provides the virtualization capabilities, upon which a virtulization solution can be built.

...It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko. - http://www.linux-kvm.org/page/Main%5FPage

Typically, a KVM-based solution involves the combination of:

KVM - for access to the CPU's virtualization capabilities
QEMU - which provides the emulation bits, such as BIOS/UEFI
additional libraries and tools - such as libvirt and Virtual Machine Manager

QEMU

QEMU supports two native disk formats (there are other formats as well, like VDI):

Raw

Raw disk image format (default). This format has the advantage of
being simple and easily exportable to all other emulators. If your
file system supports holes (for example in ext2 or ext3 on Linux or NTFS on Windows), then only the written sectors will reserve space. - man qemu-img

Qcow/2

QEMU image format, the most versatile format. Use it to have
smaller images (useful if your filesystem does not supports holes,
for example on Windows), optional AES encryption, zlib based
compression and support of multiple VM snapshots. - man qemu-img

libvirt

libvirt provides a number of ways to store disk images:

Directory backend
Local filesystem backend
Network filesystem backend
Logical backend - LVM but without thin-provisioning :(
Disk backend
iSCSI backend
SCSI backend
Multipath backend
RBD (RADOS Block Device) backend
Sheepdog backend
Gluster backend
ZFS backend
Virtuozzo storage backend

Take a look at those and see which, if any, meet your requirements. See https://libvirt.org/storage.html, and better yet https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/chap-Virtualization_Administration_Guide-Storage_Pools-Storage_Pools.html

Related Solutions

Securely delete files on btrfs filesystem

Secure deletion is a tough proposition on any filesystem. Unless the filesystem is very peculiar and guarantees that there aren't other copies of the file lying around, you need to clear all the free space on the device. While you are more likely to find many bits of the file on copy-on-write filesystems, even more “static” filesystems don't have this guarantee in practice, because many files get edited, so there are bit from former versions of the file lying around.

Note that erasing with zeroes is as good as erasing with random bytes, and you don't need multiple passes. Erasing with zeroes left residual data that could be partially recovered in lab conditions with 1980's hard disk technologies; this is no longer applicable today. See Why is writing zeros (or random data) over a hard drive multiple times better than just doing it once?

You can get rid of the cleartext confidential data by encrypting everything on the disk. Set up an ecryptfs volume over that filesystem and move all your (confidential) files to it. Then overwrite all the unused space of the filesystem. You can erase most of it by filling the filesystem with cat /dev/zero >zero. There may still be some information left in incomplete blocks (blocks that contain the last chunk of a file, followed by some garbage — which could be leftovers from a confidential file). To ensure that there are no incomplete blocks, move everything on the filesystem to ecryptfs (ecryptfs's files use whole blocks, at least on typical setups where blocks are 4kB). Make sure to apply this to all volumes and to erase all snapshots containing plaintext confidential data.

There may still be some information left in the journal. I don't know how to scrub that.

On SSD, due to block reallocation, there may be data left that can't be read by normal software means but may be recovered by hacking the firmware or with physical access. There your only recourse is a full wipe of the SSD.

Qcow2 snapshots – snapshot for disk vda unsupported for storage type raw

There is an error in your libvirt config file:

<driver name='qemu' type='raw'/>

This line must be:

<driver name='qemu' type='qcow2'/>

I just tested this conflict. If you create a qcow2 image file (which can be tested with qemu-img info /home/$USER/kvm/domain.name.img) and configure the libvirt domain for raw then I had expected to get an error message. Instead the qcow2 image file is overwritten by an raw image file.

So you have to shut down the VM, change the configuration, convert the image file (qemu-img convert -f raw -O qcow2 [input filename] [output filename]), and restart the VM.