Replace text quickly in very large file

large filesreplacesed

I have 25GB text file that needs a string replaced on only a few lines. I can use sed successfully but it takes a really long time to run.

sed -i 's|old text|new text|g' gigantic_file.sql

Is there a quicker way to do this?

Best Answer

You can try:

sed -i '/old text/ s//new text/g' gigantic_file.sql

From this ref:

OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to large input files or slow processors or hard disks), substitution will be executed more quickly if the "find" expression is specified before giving the "s/.../.../" instruction.

Here is a comparison over a 10G file. Before:

$ time sed -i 's/original/ketan/g' wiki10gb
real    5m14.823s
user    1m42.732s
sys     1m51.123s

After:

$ time sed -i '/ketan/ s//original/g' wiki10gb
real    4m33.141s
user    1m20.940s
sys     1m44.451s

Related Solutions

Substituting strings in a very large file

With awk you can avoid reading huge amount of text at once:

awk -vRS='http://' -vORS='\nhttp://' 1 urls.txt > urlsperline.txt

The success may depend on the used awk implementation. For example gawk works fine, but mawk crashes.

How to use sed or ex to replace a block (multi-line code) with new block of text (code)

I suggest using the change command (which is essentially a delete coupled with an append, though the append is only applied for the last line in the range which is exactly what you want here):

sed -i '/marker1/,/marker2/c\
New text 1\
New text 2' filename

Here using GNU sed's syntax for in-place editing (-i). That c command is otherwise standard and portable. GNU sed supports:

sed '/marker1/,/marker2/cNew text 1\
New text 2' filename

as a non-standard extension.

Newline and backslash characters must be escaped (with backslash) in the replacement text.

Best Answer

Related Solutions

Substituting strings in a very large file

How to use sed or ex to replace a block (multi-line code) with new block of text (code)

Related Question