How to use Bash to find 2 bytes in a binary file, increase their values, and replace

binaryconversionreplacetext processing

I'm trying to find two bytes inside binary file, then increase value of those two bytes and replace them inside file. Those two bytes are on positions 0x82-0x83. For now on I have successfully extracted those two bytes using this:

#!/usr/bin/env bash
BYTES=$(tail -c +131 "$1" | head -c 2)

Those bytes have value: 1B 1F. I'm stuck with:

How to convert bytes to integer? It should be 6943 decimal.
How to append / echo binary data to file
How to write increased bytes inside file on positions 0x82-0x83. I could use head -c 130 original.bin >> new_file.bin && magic_command_writing_bytes_to_file >> new_file.bin && tail -c +133 original.bin, but there must be better way.

I could do that in PHP, it should be easier, but I'm interested how to do this in bash.

Best Answer

Testing with this file:

$ echo hello world > test.txt
$ echo -n $'\x1b\x1f' >> test.txt
$ echo whatever >> test.txt
$ hexdump -C test.txt 
00000000  68 65 6c 6c 6f 20 77 6f  72 6c 64 0a 1b 1f 77 68  |hello world...wh|
00000010  61 74 65 76 65 72 0a                              |atever.|
$ grep -a -b --only-matching $'\x1b\x1f' test.txt 
12:

So in this case the 1B 1F is at position 12.

Convert to integer (there is probably an easier way)

$ echo 'ibase=16; '`xxd -u -ps -l 2 -s 12 test.txt`  | bc
6943

And the reverse:

$ printf '%04X' 6943 | xxd -r -ps | hexdump -C
00000000  1b 1f                                             |..|
$ printf '%04X' 4242 | xxd -r -ps | hexdump -C
00000000  10 92                                             |..|

And putting it back in the file:

$ printf '%04X' 4242 | xxd -r -ps | dd of=test.txt bs=1 count=2 seek=12 conv=notrunc
2+0 records in
2+0 records out
2 bytes (2 B) copied, 5.0241e-05 s, 39.8 kB/s

Result:

$ hexdump -C test.txt
00000000  68 65 6c 6c 6f 20 77 6f  72 6c 64 0a 10 92 77 68  |hello world...wh|
00000010  61 74 65 76 65 72 0a                              |atever.|

Related Solutions

Bash – How to use bash script to read binary file content

If you want to stick with shell utilities, you can use head to extract a number of bytes, and od to convert a byte into a number.

export LC_ALL=C    # make sure we aren't in a multibyte locale
n=$(head -c 1 | od -An -t u1)
string=$(head -c $n)

However, this does not work for binary data. There are two problems:

Command substitution $(…) strips final newlines in the command output. There's a fairly easy workaround: make sure the output ends in a character other than a newline, then strip that one character.
```
string=$(head -c $n; echo .); string=${string%.}
```
Bash, like most shells, is bad at dealing with null bytes. As of bash 4.1, null bytes are simply dropped from the result of the command substitution. Dash 0.5.5 and pdksh 5.2 have the same behavior, and ATT ksh stops reading at the first null byte. In general, shells and their utilities aren't geared towards dealing with binary files. (Zsh is the exception, it's designed to support null bytes.)

If you have binary data, you'll want to switch to a language like Perl or Python.

<input_file perl -e '
  read STDIN, $c, 1 or die $!;    # read length byte
  $n = read STDIN, $s, ord($c);   # read data
  die $! if !defined $n;
  die "Input file too short" if ($n != ord($c));
  # Process $s here
'

<input_file python -c '
  import sys
  n = ord(sys.stdin.read(1))      # read length byte
  s = sys.stdin.read(n)           # read data
  if len(s) < n: raise ValueError("input file too short")
  # Process s here
'

Shell: How to read the bytes of a binary file and print as hexadecimal

Use hexdump(1)

$ hexdump -x /usr/bin/hexdump 
0000000    feca    beba    0000    0300    0001    0700    0080    0300
0000010    0000    0010    0000    5080    0000    0c00    0000    0700
0000020    0000    0300    0000    00a0    0000    b06f    0000    0c00
0000030    0000    1200    0000    0a00    0100    0010    0000    107c
0000040    0000    0c00    0000    0000    0000    0000    0000    0000
0000050    0000    0000    0000    0000    0000    0000    0000    0000

...

Best Answer

Related Solutions

Bash – How to use bash script to read binary file content

Shell: How to read the bytes of a binary file and print as hexadecimal

Related Question