Linux dd – How to Prevent dd’s Progress from Being Meaningless

dd

When running dd command to copy an ISO on Linux, I get a single progress print that stays open for a long time (many minutes). Then another at the end.

The problem seems to be that a very large cache is being used which confuses dd 's output.

sudo dd bs=4M if=my.iso of=/dev/sdc status=progress

Output (first line shows for a long time).

1535115264 bytes (1.5 GB, 1.4 GiB) copied, 1.00065 s, 1.5 GB/s
403+1 records in
403+1 records out
1692844032 bytes (1.7 GB, 1.6 GiB) copied, 561.902 s, 3.0 MB/s

Is there a way to prevent this from happening so the progress output is meaningful?

Best Answer

From the first line we can tell dd has read and written 1.5GB in one second. Even an SSD can't write that fast.

What happened is that the /dev/sdc block device accepted it (writeback), but didn't send it to disk but buffered it and started writing to disk at the rate the disk can take it. Something like 3MiB/s.

The system can't buffer data indefinitely like that, there's only so much data it will accept to hold in that non-committed dirty state. So after a while (in your case, after more than 1.5GB have been written but less than 2 seconds have passed (as progress lines are written every second)), dd's write() system call will block until the data has been flushed to the disk (during which it cannot write progress messages). When it gets through, dd can send the few extra missing megabytes, and that happens within less than a second, so you get only one extra progress line.

To see a different behaviour, you could force the writes to be synchronous, that is not to return unless the data has been committed to disk. For instance by using oflag=sync or oflag=dsync or oflag=direct (not that I would advise doing that though).

Related Question