Linux – Has ‘sync’ been proved to be necessary when writing to a block device with ‘dd’

block-devicecacheddlinuxsynchronization

Every time I've written a file to an empty raw block device, e.g.

# dd if=image.iso of=/dev/sdb status=progress

I've never used any type of sync (i.e. sync; conv=fsync; conv=fdatasync; oflag=sync; oflag=dsync).

I've noticed that dd doesn't ever exit until all writing has finished.

I always verify this using Conky's I/O facility and grep Dirty /proc/meminfo. Also, the checksum of the device always matches that of the file that was written to it. So I am always 100% sure that the whole file was written to the device.

I've written files to an ext4 volume to compare. For example using:

$ dd if=/dev/urandom of=~/file bs=1M count=50 iflag=fullblock

When writing to an ext4 volume, after dd exits there is always a delay of about 20 seconds before the data is actually written to the disk.

Many people advocate using the sync command after the dd command, or including one of the several sync options in the dd command when writing to a block device. E.g. here and here. However, I've not known anyone actually prove that it is necessary.

One of the comments on this page is:

sync is pointless here [i.e. writing directly to /dev/sdX]. It only affects file system operations.

Five people have upvoted this comment and it is consistent with my experience.

So when writing to a block device, are the any circumstances in which dd will exit before all writing has completely finished? Has this actually ever happened to anyone?

What about other writing options, such as cp and cat? Can they exit before writing to a block device has finished?

Best Answer

It's about time to answer this.

For such a long time, I've been thinking that only writes to filesystems were cached, not block devices, only to discover that I was wrong : writes to block devices are cached.

Strictly speaking I don't have an official source on this, and will update my answer as I get one~~, but you can find some info here : linux - Block device cache v.s. a filesystem - Unix & Linux Stack Exchange~~. Sorry, I just noticed you saw that question too !

In fact, I had the problem of dd'ing an image to a USB stick, eject it after completion and then discover it was corrupted. It shouldn't have behaved like that since it seems a complete sync is done on the last close() on the device, hence I suppose either I got data corruption on my device, or I missed a process that still had the block device opened, but anyway I'm not taking chances any more.

So yes, adding sync options to dd when writing to a block device doesn't seem to be an urban legend. I think conv=fdatasync should be enough and best performing (no file metadata is added after all, and only syncing at the end of the process is really needed), but the extremists among us may prefer oflag=sync (complete data + metadata sync after each write). Check man 2 open and man 2 fdatasync for more insight.

According to what I've read (but nothing official as of now, unfortunately), cp and cat to a block device can indeed exit before sync if some other process still has the block device open, unless explicitly requested with some SYNC flags when the device is open()ed or with a sync system call such as fdatasync(), fsync() or more exaggeratingly sync(), which none of these commands does...

Related Solutions

GNU/Linux – Overlay Block Device and Stackable Block Device

You can do that with the device mapper and its snapshot target.

Basically, you'd do the same as what LVM does when you create a writable snapshot.

dev=/dev/read-only-device
ovl=/path/to/overlay.file
newdevname=newdevice
size=$(blockdev --getsz "$dev")

loop=$(losetup -f --show -- "$ovl")
printf '%s\n' "0 $size snapshot $dev $loop P 8" |
  dmsetup create "$newdevname"

Then you can access the overlayed device as /dev/mapper/newdevice.

If you also need access to the original device at the same time, you can do:

printf '%s\n' "0 $size snapshot-origin $dev" |
  dmsetup create originaldevice

And access it over /dev/mapper/originaldevice.

You can write to that device, then in addition to the chunks written to the snapshot device, the overlay file will contain a copy of the chunks that have been overwritten when writing to the snapshot-origin.

The overlay file can be a sparse file. (for instance, create it as truncate -s10G the-file), and doesn't have to be as large as the original device. You can tell how full it is with dmsetup status "$newdevname".

Note: There are size and contents reqirements on a snapshot device.

How to Use `dd` to Right-Shift Data Blocks

Without a buffer, you could go backwards, one block at a time.

for i in $(seq 100 -1 0)
do
    dd if=/dev/thing of=/dev/thing \
       bs=1M skip=$i seek=$(($i+2)) count=1
done

Please note that this example is dangerous due to lack of error checking.

It's also slow due to the amount of dd calls. If you have memory to spare, you could use a larger blocksize.

With a buffer, beware pitfalls. It is not sufficient to guarantee a 100% prefill. What you need is a minimum fill throughout the entire process. The buffer must never ever drop below 2M because otherwise you will have overwritten your yet-to-be-read data again.

So while in theory you could do without any kind of buffer and just chain dd:

dd if=/dev/thing bs=1M | \
dd bs=1M iflag=fullblock | \
dd bs=1M iflag=fullblock | \
dd of=/dev/thing bs=1M seek=2

In practice this does not work reliably because there is no guarantee the first dd manages to keep reading data, while the last dd (with 2M of "buffer" in between) is already writing.

You can increase your chances considerably by making the in between buffer considerably larger, but even so, it's not reliable.

Unfortunately I do not know a good buffer program with minimum fill property. You need one that stops output as long as there is less than your safety margin within the buffer.

Best Answer

Related Solutions

GNU/Linux – Overlay Block Device and Stackable Block Device

How to Use `dd` to Right-Shift Data Blocks

Related Question