“write-once archive”: ext2 vs ext4^has_journal vs

backupext2ext4filesystemsjournaling

summary

Suppose one is setting up an external drive to be a "write-once archive": one intends to reformat it, copy some files that will (hopefully) never be updated, then set it aside until I need to read something (which could be a long while or never) from the archive from another linux box. I also want to be able to get as much filespace as possible onto the archive; i.e., I want the filesystem to consume as little freespace as possible for its own purposes.

specific question 1: which filesystem would be better for this usecase: ext2, or ext4 without journaling?

Since I've never done the latter before (I usually do this sort of thing with GParted), just to be sure:

specific question 2: is "the way" to install journal-less ext4 mke2fs -t ext4 -O ^has_journal /dev/whatever ?

general question 3: is there a better filesystem for this usecase? or Something Completely Different?

details

I've got a buncha files from old projects on dead boxes (which will therefore never be updated) saved on various external drives. Collectively size(files) ~= 250 GB. That's too big for DVDs (i.e., would require too many–unless I'm missing something), and I don't have a tape drive. Hence I'm setting up an old USB2 HFS external drive to be their archive. I'd prefer to use a "real Linux" filesystem, but would also prefer a filesystem that

consumes minimum space on the archive drive (since it's just about barely big enough to hold what I want to put on it.
will be readable from whatever (presumably Linux) box I'll be using in future.

I had planned to do the following sequence with GParted: [delete old partitions, create single new partition, create ext2 filesystem, relabel]. However, I read here that

recent Linux kernels support a journal-less mode of ext4
which provides benefits not found with ext2

and noted the following text in man mkfs.ext4

"mke2fs -t ext3 -O ^has_journal /dev/hdXX"
will create a filesystem that does not have a journal

So I'd like to know

Which filesystem would be better for this usecase: ext2, or ext4 without journaling?
Presuming I go ext4-minus-journal, is the commandline to install it mke2fs -t ext4 -O ^has_journal /dev/whatever ?
Is there another, even-better filesystem for this usecase?

Best Answer

I don't agree with the squashfs recommendations. You don't usually write a squashfs to a raw block device; think of it as an easily-readable tar archive. That means you would still need an underlaying filesystem.

ext2 has several severe limitations that limit its usefulness today; I would therefore recommend ext4. Since this is meant for archiving, you would create compressed archives to go on it; that means you would have a small number of fairly large files that rarely change. You can optimize for that:

specify -I 128 to reduce the size of individual inodes, which reduces the size of the inode table.
You can play with the -i option too, to reduce the size of the inode table even further. If you increase this value, there will be less inodes created, and therefore the inode table will also be smaller. However, that would mean the filesystem wastes more space on average per file. This is therefore a bit of a trade-off.
You can indeed switch off the journal with -O ^has_journal. If you go down that route, though, I recommend that you set default options to mount the filesystem read-only; you can do this in fstab, or you could use tune2fs -E mount_opts=ro to record a default in the filesystem (you cannot do this at mkfs time)
you should of course compress your data into archive files, so that the inode wastage isn't as bad a problem as it could be. You could create squashfs images, but xz compresses better, so I would recommend tar.xz files instead.
You could also reduce the number of reserved blocks with the -m option to either mkfs or tune2fs. This sets the percentage (set to 5 by default) which is reserved for root only. Don't set it to zero; the filesystem requires some space for efficient operation.

Related Solutions

Disabling journal vs data=writeback in ext4 file system

The two are in no way equivalent. Disabling the journal does exactly that: turns journaling off. Setting the journal mode to writeback, on the other hand, turns off certain guarantees about file data while assuring metadata consistency through journaling.

The data=writeback option in man(8) mount says:

Data ordering is not preserved - data may be written into the main filesystem after its metadata has been committed to the journal. This is rumoured to be the highest- throughput option. It guarantees internal filesystem integrity, however it can allow old data to appear in files after a crash and journal recovery.

Setting data=writeback may make sense in some circumstances when throughput is more important than file contents. Journaling only the metadata is a compromise that many filesystems make, but don't disable the journal entirely unless you have a very good reason.

Reliable way to detect ext2 or ext3 or ext4

After looking at the code for various utilities and the kernel code for some time, it does seem that what @Hauke suggested is true - whether a filesystem is ext2/ext3/ext4 is purely defined by the options that are enabled.

From the Wikipedia page on ext4:

Backward compatibility

ext4 is backward compatible with ext3 and ext2, making it possible to mount ext3 and ext2 as ext4. This will slightly improve performance, because certain new features of ext4 can also be used with ext3 and ext2, such as the new block allocation algorithm.

ext3 is partially forward compatible with ext4. That is, ext4 can be mounted as ext3 (using "ext3" as the filesystem type when mounting). However, if the ext4 partition uses extents (a major new feature of ext4), then the ability to mount as ext3 is lost.

As most probably already know, there is similar compatibility between ext2 and ext3.

After looking at the code which blkid uses to distinguish different ext filesystems, I was able to turn an ext4 filesystem into something recognised as ext3 (and from there to ext2). You should be able to repeat this with:

truncate -s 100M testfs
mkfs.ext4 -O ^64bit,^extent,^flex_bg testfs <<<y
blkid testfs
tune2fs -O ^huge_file,^dir_nlink,^extra_isize,^mmp testfs
e2fsck testfs
tune2fs -O metadata_csum testfs
tune2fs -O ^metadata_csum testfs
blkid testfs
./e2fsprogs/misc/tune2fs -O ^has_journal testfs
blkid testfs

First blkid output is:

testfs: UUID="78f4475b-060a-445c-a5d2-0f45688cc954" SEC_TYPE="ext2" TYPE="ext4"

Second is:

testfs: UUID="78f4475b-060a-445c-a5d2-0f45688cc954" SEC_TYPE="ext2" TYPE="ext3"

And the final one:

testfs: UUID="78f4475b-060a-445c-a5d2-0f45688cc954" TYPE="ext2"

Note that I had to use a new version of e2fsprogs than was available in my distro to get the metadata_csum flag. The reason for setting, then clearing this was because I found no other way to affect the underlying EXT4_FEATURE_RO_COMPAT_GDT_CSUM flag. The underlying flag for metadata_csum (EXT4_FEATURE_RO_COMPAT_METADATA_CSUM) and EXT4_FEATURE_RO_COMPAT_GDT_CSUM are mutually exclusive. Setting metadata_csum disables EXT4_FEATURE_RO_COMPAT_GDT_CSUM, but un-setting metadata_csum does not re-enable the latter.

Conclusions

Lacking a deep knowledge of the filesystem internals, it seems either:

Journal checksumming is meant to be a defining feature of a filesystem created as ext4 that you are really not supposed to disable and that fact that I have managed this is really a bug in e2fsprogs. Or,
All ext4 features were always designed to be disabled and disabling them does make the filesystem to all intents an purposes an ext3 filesystem.

Either way a high level of compatibility between the filesystems is clearly a design goal, compare this to ReiserFS and Reiser4 where Reiser4 is a complete redesign. What really matters is whether the features present are supported by the driver that is used to mount the system. As the Wikipedia article notes the ext4 driver can be used with ext3 and ext2 as well (in fact there is a kernel option to always use the ext4 driver and ditch the others). Disabling features just means that the earlier drivers will have no problems with the filesystem and so there are no reasons to stop them from mounting the filesystem.

Recommendations

To distinguish between the different ext filesystems in a C program, libblkid seems to be the best thing to use. It is part of util-linux and this is what the mount command uses to try to determine the filesystem type. API documentation is here.

If you have to do your own implementation of the check, then testing the same flags as libblkid seems to be the right way to go. Although notably the file linked has no mention of the EXT4_FEATURE_RO_COMPAT_METADATA_CSUM flag which appears to be tested in practice.

If you really wanted to go the whole hog, then looking at for journal checksums might be a surefire way of finding if a filesystem without these flags is (or perhaps was) ext4.

Update

It is actually somewhat easier to go in the opposite direction and promote an ext2 filesystem to ext4:

truncate -s 100M test
mkfs.ext2 test
blkid test
tune2fs -O has_journal test
blkid test
tune2fs -O huge_file test
blkid test

The three blkid ouputs:

test: UUID="59dce6f5-96ed-4307-9b39-6da2ff73cb04" TYPE="ext2"

test: UUID="59dce6f5-96ed-4307-9b39-6da2ff73cb04" SEC_TYPE="ext2" TYPE="ext3"

test: UUID="59dce6f5-96ed-4307-9b39-6da2ff73cb04" SEC_TYPE="ext2" TYPE="ext4"

The fact that ext3/ext4 features can so easily by enabled on a filesystem that started out as ext2 is probably the best demonstration that the filesystem type really is defined by the features.