How long can file system writes be cached with ext4

ext4filesystems

A while ago, there has been some discussion about ext4 potentially leaving empty files after an unclean unmount, summed up pretty well in this article. Basically, because of delayed allocation, writes can be kept in the write cache for a much longer time than the default commit interval of the ext journal (5 seconds).

The problems seem to have been fixed in a patch that forces block allocation in certain situations, thereby forcing the data to disk after at most 5 seconds by default.

I'm wondering what happens when an application overwrites existing parts of a file, without truncating or appending the file itself. Will that be forced to disk within 5 seconds as well?

It seems like a different situation than appending to a file: when appending, the file size changes, which is a metadata change; therefore, a journal commit will be necessary within 5 seconds, and because of data=ordered, the data will have to be written before that because of security concerns (otherwise parts of deleted files of other users could show up for the owner of the appended file).

When just overwriting file data, there is no reason why the data write should have to happen before the metadata journal commit, as the old data belongs to the same user as the new one. So does the write happen before the commit anyway, or can it be delayed longer than the journal commit interval? If so, how long?

Update:
I know that all this is irrelevant when doing the right thing, that is, using fsync(). (This was the main reason for all the discussion about ext4 and data loss – the problem only concerned applications not fsync()ing, or not at the right moments.) I'm not writing my own application, I'm asking because I don't know whether all of my applications do the right thing, and I want to know an approximate timeframe for such "dangerous" writes. The reason for asking is that my graphics driver causes kernel panics regularly, and I want to know whether I have to worry about more than the last 5 seconds of data writes.

Best Answer

You can set the commit interval to a custom value which, I believe, can be as high as a 32-bit unsigned integer number of seconds; so about 4 billion seconds, or 136 years. This is available through the commit mount option, which you can place into effect as follows (this is merely an example; you can also set this in fstab):

mount /dev/sda1 -t ext4 -o rw,data=writeback,nobh,commit=12345678

The commit interval is not based on any type of condition like whether the data is appended or overwriting existing data or whatever. The commit mount option (which defaults to 5 seconds if you don't supply the mount option at all) is equivalent to doing something like this in a bash shell:

#!/bin/bash
while :
do
    echo "Syncing all uncommitted data and journal to disk"
    sync
    sleep 5
done

Don't confuse data=ordered and this global filesystem sync interval ("commit interval" is perhaps a less meaningful term for those of us who understand the functionality of the command line program sync, in which case it may be better named the "sync interval"). data=ordered is about the order in which data and metadata are updated (where data=writeback is "less safe / faster" and data=journal is "more safe / slower"). commit=12345678 is about the frequency with which the filesystem driver itself forces a FULL sync of ALL dirty data/journal/metadata/whatever to the physical media. And you can most certainly set it to 136 years if you want, and mount with data=writeback,nobh and programs that don't call fsync() or sync() will have dirty pages sitting in RAM for... several lifetimes.

Update: Based on your context in your question edit, I'd say that you should run your filesystem with mount options data=journal,commit=1 or even with the sync mount option, until you are able to resolve your graphics driver kernel panics. This will maintain maximum data integrity but at the cost of performance. You'll especially want to do this if you are frequently writing data to disk that you can't afford to lose, and that's doubly important if you don't "trust" the apps you're using to employ fsync() appropriately.

Source: here and personal experience

Related Question