Shell – How to delete part of a binary file without copying

ddfilesshell

I need to delete the first 2 bytes of a 6MB file. However, this is an embedded Linux with only 32 Mbytes RAM and less than 1 MB free flash memory.

I tried using dd, as:

1 – # dd bs=1 skip=2 count=1022 if=input of=ouput_1

2 – # dd bs=1024 skip=1 if=input of=ouput_2

3 – # rm -rf input

4 – # (dd if=ouput_1 ; dd if=ouput_2) > ouput

5 – # rm -rf ouput_1 ouput_2

With all files under the /tmp (mounted as tmpfs on RAM), my problem is that just before lines 3 and 5, the memory needed is 12 Mbyte (2x6MB), and the process sometimes fail and gives an "Not enough memory" error.

Is there a way I can remove the first 2 bytes without allocating twice the size of the file ? Can I use dd (or any other) to cut a binary file 'in place' ?

Best Answer

I think this should work:

$   # Create test file
$ echo "Hello, World" > h.data
$
$   # Move contents up by 2 bytes
$   # Note if= and of= are the same for in-place editing
$ dd bs=2 if=h.data skip=1 seek=0 conv=notrunc of=h.data
5+1 records in
5+1 records out
11 bytes (11 B) copied, 0.000598796 s, 18.4 kB/s
$
$   # Note 11 bytes were moved above
$   # Truncate the file after byte 11
$ dd bs=11 if=h.data skip=1 seek=1 count=0 of=h.data
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000338852 s, 0.0 kB/s
$
$   # Display edited file:
$ cat h.data
llo, World
$

Wrapping this all up in a script you could have something like this:

#!/bin/bash

size=$(stat -c %s "$2")
dd bs=$1 if="$2" skip=1 seek=0 conv=notrunc of="$2"
dd bs=$((size - $1)) if="$2" skip=1 seek=1 count=0 of="$2"

Call this as:

./truncstart.sh 2 file.dat

Where 2 is the number of bytes to delete from the beginning of file.data

As @Gilles points out, this solution is not robust in case of unplanned outage, which could occur part-way through dd's processing; in which case the file would be corrupted.

Related Question