How large is the img that dd creates

ddhard-disk

This is the first time I use the dd command.
I execute:

dd if=/dev/sdb2 of=/mnt/sdc1/Hdd1.img bs=512 conv=noerror,sync

Where the sdb is the destroyed hdd (size: 500 GB).
I copying the partition sdb2 into an image. I've done it 6(!!) days. the img size is about 640 GB and still counting (i.e: it hasn't finished yet…). 6 days it's printing the copied data details (which byte it's copied to where) and it's not stopping.

Is it normal? how is it possible that the img size is larger than the whole destroyed hdd size? and when it suppose to finishing?

Best Answer

By doing the copy 512 bytes at a time you are doing lots and lots of reads and writes. About a trillion, actually, if you do the math. You've also asked for sync [EDIT: this is not oflag=sync so the next statement is invalid], which means to wait for each write to actually make it out to disk before that write can return. Let's say your disk is pretty speedy so each write takes 2ms.

500gb / 512 bytes * 2ms = 22.6 days.

Wow, ~~trillions~~ billions of milliseconds added up fast, didn't they?

[EDIT: while that was certainly a fun bit of math, it's not accurate since oflag=sync wasn't used. The delays are more likely due to repeatedly reading bad sectors and those associated timeouts. The below dd_rescue approach should help quite a bit. Using plain dd with a larger block size might help, but not as much since it can't adapt its read size and won't skip over massive damage.]

If you use a larger block size and/or skipped the sync it will run MUCH faster:

# dd if=/dev/sdb2 of=/sdb2-image.img bs=1024k

If you're concerned about read errors on the sdb2 image read, use dd_rescue with the -A option to write out a block of zeroes instead of skipping that write. Skipping blocks with errors entirely can lead to problems when certain filesystem structures appear at different offsets from the start than they were originally. It's better to just have some unexpected zeroes. For example:

# dd_rescue -A /dev/sdb2 /sdb2-image.img

This will start out reading large blocks of data at once and only reduces it when it starts hitting errors.

EDIT: to directly answer the question, as suggesed by Micheal Johnson, when using conv=noerror,sync on dd or -A on dd_rescue, your image will end up the exact same size as your source. This is because every read will generate an identically sized write. Some versions of dd may keep running long past the end of the device since they ignore the end-of-file "error" per your conv=noerror request. I don't think Linux does this, but it's something to watch out for if your image seems to be getting larger than the source.

Update

fullblock option added above as per @Gilles answer. At first I thought that it might be implied by count_bytes, but this is not the case.

The issues mentioned are a potential problem below, if dds read/write calls are interrupted for any reason then data will be lost. This is not likely in most cases (odds are reduced somewhat since we are reading from a file and not a pipe).

Using a dd without the skip_bytes and count_bytes options is more difficult:

in_file=1tb

start=12345678901
end=19876543212
block_size=4096

copy_full_size=$(( $end - $start ))
copy1_size=$(( $block_size - ($start % $block_size) ))
copy2_start=$(( $start + $copy1_size ))
copy2_skip=$(( $copy2_start / $block_size ))
copy2_blocks=$(( ($end - $copy2_start) / $block_size ))
copy3_start=$(( ($copy2_skip + $copy2_blocks) * $block_size ))
copy3_size=$(( $end - $copy3_start ))

{
  dd if="$in_file" bs=1 skip="$start" count="$copy1_size"
  dd if="$in_file" bs="$block_size" skip="$copy2_skip" count="$copy2_blocks"
  dd if="$in_file" bs=1 skip="$copy3_start" count="$copy3_size"
}

You could also experiment with different block sizes, but the gains won't be very dramatic. See - Is there a way to determine the optimal value for the bs parameter to dd?

Linux – Create Empty Image with 4096 Byte Sectors Using dd

The bs given to dd just tells how large the buffer should be during creating the file. In the end, the file consists of nothing but zero-bytes, there is no information about alignment.

You have to use the specific parameter to fdisk, which is -b, as per the man-page of fdisk(8):

  -b, --sector-size sectorsize
          Specify  the  sector  size  of  the  disk.   Valid values are 512,    1024, 2048, and 4096.  (Recent kernels know the sector size.  Use this option only on old kernels or to override the kernel's
          ideas.)  Since util-linux-2.17, fdisk differentiates between logical and physical sector size.  This option changes both sector sizes to sectorsize.

Best Answer

Related Solutions

File Reading – How to Read the Middle of a Large File

Update

Linux – Create Empty Image with 4096 Byte Sectors Using dd

Related Question