Copying raw partition sequentially, in chunks, with error recovery

backupddddrescuepartitioning

Normally if I want to copy a raw partition /dev/sdaX (with i/o errors when reading), I would just go with ddrescue <src> <dst>.

The case here is that I'd like to be able to copy this partition in smaller chunks (e.g. 1GB partition in 10*100MB chunks), but not all at once, and glue them together afterwards. Imagine this as a scenario where you want to copy that 1GB partition from PC_A to PC_B but you only have a 100MB usb stick available.

Is this technically possible? Would ddrescue be able to do that, if so, how?

Best Answer

This is technically possible. The filesystem on the USB stick must support sparse files for my method to work.

To tell ddrescue to recover a part of a file, use -m:

-m file
--domain-mapfile=file
Restrict the rescue domain to the blocks marked as finished in the mapfile file. […]

(source)

You need to prepare many domain-mapfiles, each describing one rescue domain, one chunk. Altogether they should cover the entire device. Read about the mapfile structure. The 0th domain-mapfile chunk00.map will be:

# status line is a formality
0 + 1
# the below line describes the chunk
# pos size      status
0     104857600 +

This means the chunk starts at offset 0 and takes 104857600 bytes. The latter number is interpreted as decimal. ddrescue creates mapfiles with hexadecimal values (0x…) but it understands decimal. A leading zero denotes octal, so don't use leading zeros.

Alternatively you can specify these numbers with -i and -s command line options; but because we will need the exact same numbers on PC_A and PC_B, it's better to have them in a file you can copy. If you use the right values on PC_A and copy the file to PC_B then you will use the right values on PC_B. Typing commands with -i and -s on both systems independently is more error-prone.

104857600 bytes is 100 mebibytes. I did not use 100000000 (100 megabytes) because we're going to use either -b 512 or -b 4096 (depending on the physical sector size of the source device) and the chunk should take an integer number of sectors. 104857600 is a multiple of 4096 (and therefore a multiple of 512), 100000000 is not.

chunk01.map will be:

0 + 1
104857600 104857600 +

chunk02.map:

0 + 1
209715200 104857600 +

and so on. The starting position for the chunk N+1 is the starting position of the chunk N plus the size of the chunk N. In general the chunks may overlap but we don't want them to overlap.

Read the 0th chunk:

# on PC_A
ddrescue -b 512 -c 32768 -m chunk00.map /dev/sdaX /mnt/usb_PC_A/chunk00.raw common.map

You need to do this for each chunk, you need to generate chunk01.raw from chunk01.map, chunk02.raw from chunk02.map and so on. common.map should be the same mapfile each time, each invocation of ddrescue on PC_A should update the common mapfile.

Resulting chunks with non-zero offset (i.e. all chunks but chunk00.raw) will be sparse.

If the filesystem on the USB stick did not support sparse files then each chunk would be bigger than the previous one due to explicit zeros in the beginning. The last chunk would be as big as /dev/sdaX, this defies the purpose. A workaround can be to write a chunk to a local filesystem that supports sparse files, pack it (with or without compression) somehow and finally unpack on PC_B as sparse. It's way easier if the filesystem on the USB stick supports sparse files.

After reading each chunk you should transfer the chunk to PC_B. You need to transfer chunk*.raw and its corresponding chunk*.map. If you need to copy a sparse chunk to another filesystem, use cp --sparse=always …. You don't necessarily need to copy though; building the final image can be done by reading directly from the USB stick.

Before transferring the 0th chunk, make sure the final image final.raw on PC_B is empty or nonexistent. Transfer the 0th chunk from the USB:

# on PC_B
ddrescue -b 512 -c 32768 -m /mnt/usb_PC_B/chunk00.map /mnt/usb_PC_B/chunk00.raw final.raw

Do this with all chunks as they come, writing to the same final.raw; just remember to use the right chunk*.map each time. Here ddrescue reads regular files, we don't expect read errors, there's no need for a mapfile.

After transferring all the chunks this way, final.raw on PC_B should be complete. The file common.map on PC_A is the corresponding mapfile. It's as if you did

ddrescue -b 512 -c 32768 /dev/sdaX final.raw common.map

on PC_A, except final.raw is now on PC_B.

Notes:

  • In case of read errors on PC_A, a chunk may be smaller than you expect. See "conclusions" in this answer of mine (where "chunk" means "one or more sectors", "a fragment", it's different than our chunk):

    Unreadable chunk [=fragment] at the very end of the input file doesn't contribute to the overall size of the output file.

    If our last chunk is smaller then final.raw on PC_B will be smaller than /dev/sdaX on PC_A. It would be exactly like this if you let ddrescue write to final.raw on PC_A in one run.

  • In my tests a chunk that starts within /dev/sdaX and reaches beyond the end of sdaX generates adequately smaller chunk*.raw file. A chunk that starts beyond the end of sdaX makes ddrescue on PC_A complain and fail. This means you don't need to pay attention to the size of sdaX, you don't need to define the last chunk accurately. While building chunk*.map files you can increase the starting position by your desired size each time, until some chunk*.raw is created smaller because of the end of sdaX. Then you try to ddrescue the next chunk, get Can't start reading at pos … . Input file is only … bytes long. from the tool and this way you find out the whole sdX has been read. common.map should confirm this.

Related Question