Substituting strings in a very large file

sedtext processing

I have a very long series of urls with no separating character, in the same format as below:

http://example.comhttp://example.nethttp://example.orghttp://etc...

I want each URL to be on a new line. I tried to do this by replacing all instances of "http://" with "\nhttp://" using sed

sed 's_http://_\nhttp://_g' urls.txt

but a segmentation fault occurs (memory violation). I can only surmise that the sheer size of the file (it's over 100GB) is causing sed to exceed some limit.

I could split the file into several smaller files for processing, but all instances of "http://" would need to be kept intact.

Is there a better way to do this?

Best Answer

With awk you can avoid reading huge amount of text at once:

awk -vRS='http://' -vORS='\nhttp://' 1 urls.txt > urlsperline.txt

The success may depend on the used awk implementation. For example gawk works fine, but mawk crashes.

Related Solutions

Extracting lines by key from very large file

Using awk should be efficient enough - it provides builtin associative arrays, where the key lookup time is logarithmically proportional to the number of keys (of your lookup table - which is relatively small in your example).

For your input this would be:

42M * log2(1.5M) -> 42M * 20 key comparisons

(where M means 10^6)

In case your awk uses hash tables, every key lookup would only cost a constant amount of time.

An example of an efficient awk based solution (using the default field separator):

$ awk 'ARGIND == 1 { a[$1] = 1; next } a[$1] { print $0 }' keys.dat largefile.dat

Since both inputs are sorted you could write a script that would be more efficient (with a runtime scaling linearly with both input file sizes). But it would cost more time programming it.

Or you could use join which expect sorted files as input - restriction is that your key needs to be alphabetically sorted - and perhaps you have to tweak the output format. For example:

$ join -j1 keys.dat largefile.dat

Use -t to configure the field separator and -o to adjust the output format.

This should run in time linear to the input size.

Lum – Column mismatch and substituting

awk -v 'OFS=\t' 'NF == 2 { print $1, "none", $2; next } 1' input.txt > output.txt

Adjust depending on the characteristics of your input file. I assume every line with only 2 fields should have a "none" inserted. Otherwise, all other lines are just passed through unchanged (the purpose of the 1 at the end).

Best Answer

Related Solutions

Extracting lines by key from very large file

Lum – Column mismatch and substituting

Related Question