How to duplicate a file without copying its data with btrfs

btrfscpdeduplicationfile-copy

I have no experience with btrfs, but it's advertised to be able to
de-duplicate files.

In my application, I'd need to duplicate whole directory trees.

From what I learned, btrfs only de-duplicates in some post scan, not
immediately. Even just using cp doesn't seem to trigger any
de-duplication (at least, df shows an increased disk usage in the
size of the copied files).

Can I avoid moving data around altogether and tell btrfs directly to
duplicate a file at another location, essentially just cloning its
metadata?

In essence, similar to a hardlink, but with independent metadata
(permissions, mod. times, …).

Best Answer

There are two options:

cp --reflink=always
cp --reflink=auto

The second is almost always preferable to the first. Using auto means it will fallback to doing a true copy if the file system doesn't support reflinking (for instance, ext4 or copying to an NFS share). With the first option, I'm pretty sure it will outright fail and stop copying.

If you are using this as part of a script that needs to be robust in the face of non-ideal conditions, auto will serve your better.

Related Solutions

What happens if a file is modified while you’re copying it

If fileA.big is grown during the copy, the copy will include the data that was appended.

If the file is truncated shorter than where the copy is currently at, the copy will abort right where its at and the destination file will contain what was copied up to the time it aborted.

How to find data copies of a given file in Btrfs filesystem

The whole point of having a Copy-On-Write (CoW) filesystem like btrfs is that the content of multiple versions of a file can be efficiently shared. So you might see a file as a collection of ranges with contents, where the content might or might not be shared by other files. Or by other versions of the file. The implementation is more like a tree of extends, where extends may be shared.

The same mechanism which works during writing a change to a file (and therefore producing a new version of that file) is being used to do the deduplification. The implementation is described on https://github.com/g2p/bedup :

Deduplication is implemented using a Btrfs feature that allows for cloning data from one file to the other. The cloned ranges become shared on disk, saving space.

The implementation in the kernel is (for example) at http://lxr.free-electrons.com/source/fs/btrfs/ioctl.c#L2843 ; the comment makes it clear that it is not about 'reflinking' the file, but about ranges:

2843 /**
2844  * btrfs_clone() - clone a range from inode file to another
2845  *
2846  * @src: Inode to clone from
2847  * @inode: Inode to clone to
2848  * @off: Offset within source to start clone from
2849  * @olen: Original length, passed by user, of range to clone
2850  * @olen_aligned: Block-aligned value of olen, extent_same uses
2851  *               identical values here
2852  * @destoff: Offset within @inode to start clone
2853  */

So it is not the file which is reflinked, its the range which is shared. A new file could also have been constructed by sharing range with multiple files. Or being shared across volumes. Or (not sure if this is currently supported) even having the same range multiple times in the same file ;)

Therefore, no high-level tool exists to find files which share the whole file since this is a derived concept. Of course it would be possible to write support for it, but that's not the case as far as I know...

Best Answer

Related Solutions

What happens if a file is modified while you’re copying it

How to find data copies of a given file in Btrfs filesystem

Related Question