Command line tool to insert line breaks into a long string

You can use Python's csv module.

A simple example:

import csv
reader = csv.reader(open("test.csv", "r"))
for row in reader:
    for col in row:
        print col

The first thing that occurs to me when facing this type of problem is to change the record separator. In most tools, this is set to \n by default but that can be changed. For example:

1. Perl

   perl -0x3E -pe 's/<foobar>/\n$&/' file
   

   Explanation

   • -0 : this sets the input record separator to a character given its hexadecimal value. In this case, I am setting it to > whose hex value is 3E. The general format is -0xHEX_VALUE. This is just a trick to break the line into manageable chunks.
   • -pe : print each input line after applying the script given by -e.
   • s/<foobar>/\n$&/ : a simple substitution. The $& is whatever was matched, in this case <foobar>.

2. awk

   awk '{gsub(/foobar>/,"\n<foobar>");printf "%s",$0};' RS="<" file
   

   Explanation

   • RS="<" : set the input record separator to >.
   • gsub(/foobar>/,"\n<foobar>") : substitute all cases of foobar> with \n<foobar>. Note that because RS has been set to <, all < are removed from the input file (that's how awk works) so we need to match foobar> (without a <) and replace with \n<foobar>.
   • printf "%s",$0 : print the current "line" after the substitution. $0 is the current record in awk so it will hold whatever was before the <.

I tested these on a 2.3 GB, single-line file created with these commands:

for i in {1..900000}; do printf "blah blah <foobar>blah blah"; done > file
for i in {1..100}; do cat file >> file1; done
mv file1 file

Both the awk and the perl used negligible amounts of memory.