Linux – Best way to grep big binary file

data-recoverygreplinux

What is the fastest way to grep 400gb binary file?
I need one txt file from hdd dump and i know some strings from it and want to find this file in dump.

I tried to use grep -a -C 10 searchstring but grep crashes with out of memory when it tries to read large chunk of data without newline symbols. Also i would like to start searching from not from the beginning but from some point of file

Best Answer

I would use strings that way :

strings 400Gfile.bin | grep -C 10 searchstring

To start at a given offset (eg: 20G),

dd if=400Gfile.bin bs=20G skip=1 | strings | grep -C 10 searchstring

Related Solutions

Shell – Best Way to Find Multiple Strings in Large Text File

grep supports getting patterns from a file -f, and becomes more efficient if you also specify fixed strings (-F):

grep -F -f patterns.txt "//'MVS.DATASET'"

Proc Kcore – Structure of /proc/kcore on 64-bit Machine and Relation to Physical Memory

After a lot more searching I think I have convinced myself that there is no simple way to get what I want.

So, what did I end up doing? I installed LiME from github (https://github.com/504ensicsLabs/LiME)

git clone https://github.com/504ensicsLabs/LiMe
cd /LiME/src
make -C /lib/modules/`uname -r`/build M=$PWD modules

The above commands create the lime.ko kernel module. A full dump of memory can be obtained by then running:

insmod ./lime.ko "path=/root/temp/outputDump.bin format=raw dio=0"

which just inserts the kernel module and the string are the parameters specifying the output file location and format... AND IT WORKED! YAY.

Best Answer

Related Solutions

Shell – Best Way to Find Multiple Strings in Large Text File

Proc Kcore – Structure of /proc/kcore on 64-bit Machine and Relation to Physical Memory

Related Question