Remove null bytes from the end of a large file

ddfilesnull

I just backed up the microSD card from my Raspberry Pi on my PC running a Linux distro using this command:

dd if=/dev/sdx of=file.bin bs=16M

The microSD card is only 3/4 full so I suppose there's a few gigs of null bytes at the end of the tremendous file. I am very sure I don't need that. How can I strip those null bytes from the end efficiently so that I can later restore it with this command?

cat file.bin /dev/zero | dd of=/dev/sdx bs=16M

Best Answer

To create a backup copy of a disk while saving space, use gzip:

gzip </dev/sda >/path/to/sda.gz

When you want to restore the disk from backup, use:

gunzip -c /path/to/sda.gz >/dev/sda

This will likely save much more space than merely stripping trailing NUL bytes.

Removing trailing NUL bytes

If you really want to remove trailing NUL bytes and you have GNU sed, you might try:

sed '$ s/\x00*$//' /dev/sda >/path/to/sda.stripped

This might run into a problem if a large disk's data exceeds some internal limit of sed. While GNU sed has no built-in limit on data size, the GNU sed manual explains that system memory limitations may prevent processing of large files:

GNU sed has no built-in limit on line length; as long as it can malloc() more (virtual) memory, you can feed or construct lines as long as you like.

However, recursion is used to handle subpatterns and indefinite repetition. This means that the available stack space may limit the size of the buffer that can be processed by certain patterns.

Related Solutions

dd – Best Way to Remove Bytes from Start of a File

You can switch bs and skip options:

dd bs=1131 skip=1 if=filtered.dump of=trimmed.dump

This way the operation can benefit from a greater block.

Otherwise, you could try with tail (although it's not safe to use it with binary files):

tail -c +1132 filtered.dump >trimmed.dump

Finally, you may use 3 dd instances to write something like this:

dd if=filtered.dump bs=512k | { dd bs=1131 count=1 of=/dev/null; dd bs=512k of=trimmed.dump; }

where the first dd prints its standard output filtered.dump; the second one just reads 1131 bytes and throws them away; then, the last one reads from its standard input the remaining bytes of filtered.dump and write them to trimmed.dump.

File Reading – How to Read the Middle of a Large File

This is slow because of the small block size. Using a recent GNU dd (coreutils v8.16 +), the simplest way is to use the skip_bytes and count_bytes options:

in_file=1tb

start=12345678901
end=19876543212
block_size=4096

copy_size=$(( $end - $start ))

dd if="$in_file" iflag=skip_bytes,count_bytes,fullblock bs="$block_size" \
  skip="$start" count="$copy_size"

Update

fullblock option added above as per @Gilles answer. At first I thought that it might be implied by count_bytes, but this is not the case.

The issues mentioned are a potential problem below, if dds read/write calls are interrupted for any reason then data will be lost. This is not likely in most cases (odds are reduced somewhat since we are reading from a file and not a pipe).

Using a dd without the skip_bytes and count_bytes options is more difficult:

in_file=1tb

start=12345678901
end=19876543212
block_size=4096

copy_full_size=$(( $end - $start ))
copy1_size=$(( $block_size - ($start % $block_size) ))
copy2_start=$(( $start + $copy1_size ))
copy2_skip=$(( $copy2_start / $block_size ))
copy2_blocks=$(( ($end - $copy2_start) / $block_size ))
copy3_start=$(( ($copy2_skip + $copy2_blocks) * $block_size ))
copy3_size=$(( $end - $copy3_start ))

{
  dd if="$in_file" bs=1 skip="$start" count="$copy1_size"
  dd if="$in_file" bs="$block_size" skip="$copy2_skip" count="$copy2_blocks"
  dd if="$in_file" bs=1 skip="$copy3_start" count="$copy3_size"
}

You could also experiment with different block sizes, but the gains won't be very dramatic. See - Is there a way to determine the optimal value for the bs parameter to dd?