Concatenate n lines with sed

sedsorttext processing

Recently, I asked a question how to sort pairs of lines, and one of the answers suggested concatenating lines with sed, like this:

cat file.txt | sed -n 'N;s/\n//;p' | sort -t";" -k43,43n | perl -F';' -ane '$,=";";print @F[0..13],"\n";print @F[14..$#F]'

which worked brilliantly, but now my problem generalized to sorting n-tupels of lines, which I can't figure out how to do with sed.

Everything I found was either for 2 lines or ALL lines, but I need n lines (where n is 5 at the moment, but a general way would be great).

Bonus points for a nice way of rewriting the perl part to accomodate n lines, but the problem is really about the sed part.

I am also not hung on sed specifically, so if you have a nice solution using a different command line tool, please post it.

Update: Example input (n == 3)

a1;b1;c1; 
n1;m1;l1; 
d1;e1;f1;g1
n2;m2;l2;
a2;b2;c2;
d2;e2;f2;g2

Best Answer

sed -e:n -e$\bo -e'N;s/\n/&/4;to' -ebn -e:o -e'y/\n/ /' <in >out

That will concatenate 5 lines - or 1 + 4 lines - replacing each newline with a single space. However:

paste -d\  - - - - - <in >out

...would also work.

Your g sort thing could work like:

paste -d\  - - <input   |
sed 's/.*;\(.*\)/\1;&/' |
sort -t\; -k1,1         |
cut  -d\; -f2-          |
tr \  \\n

...which would be a fairly general way of doing it, though it relies on there being no spaces in the input file. it joins every two lines on a space, the copies the last ; split field to the head of each line, sorts on the first field, then cuts it away and splits the lines back out.

Related Solutions

Concatenate lines by first column by awk or sed

To get the first columns in each line using awk you can do the following:

< testfile awk '{print $1}'
aaa
aaa
aaa
www
hhh
hhh

These are your keys for the rest of the lines. So you may create a hash table, using the first column as a key and the second column of the line as the value:

< testfile awk '{table[$1]=table[$1] $2;} END {for (key in table) print key " => " table[key];}'
www => yyy
aaa => bbbNULLbbb
hhh => 111111

To get the whole rest of the line, starting with column 2, you need to collect all columns:

< testfile awk '{line="";for (i = 2; i <= NF; i++) line = line $i " "; table[$1]=table[$1] line;} END {for (key in table) print key " => " table[key];}'
www => yyy hhh NULL NULL NULL NULL 
aaa => bbb ccc ddd NULL NULL NULL NULL NULL NULL NULL NULL NULL bbb ccc    NULL NULL NULL NULL 
hhh => 111 333 yyy ooo hyy uuuioooy 111 333 yyy ooo hyy NULL

Replacing string in all files found by grep. Can’t get it to work

Typically, when you get a > in the next line after hitting, it means that one of your quotes isn't closed yet. I couldn't find that mistake in your regex. But you do not need to surround the path /var/www_data/somepath/ with single quotes. I assume there are no unusual characters in somepath?

Anyways, I tested your regex with sed. \d\w look like vim syntax for me, that's why I translated it to ascii (which always works). Also, inside of [] you do not need to escape .:

sed -r "s/'([A-Za-z0-9_-.]+)(@domain.com)'/'adsf'/g" test.dat

Indeed you can use sed or perl for your task. You don't necessarily need grep to generate a file list, unless you have GB of data. Then presorting could result in a speed benefit.

To test your regex, you could do the following:

cd /var/www_data/somepath/
sed -r 's|pattern|replace-pattern|g' a_single_file.php

When you're satisfied with the result, just add the -ibak (--in-place=bak) argument and run it on all files

find . -type f -name '*.php' -o -name '*.ini' -o name '*.conf' -o -name '*.sh' \
-exec sed -r -ibak 's|pattern|replace-pattern|g' '{}' \;

The original files are being put into <orignalname.php>.bak.

To answer your last question. For this job, grep is the tool you want, you could run it on the .bak files generated by sed above:

grep --recursive --include='*.bak' -E --files-with-matches 'pattern' . > files_fixed.txt

or, simply:

find . -type f -name '*.bak'

Best Answer

Related Solutions

Concatenate lines by first column by awk or sed

Replacing string in all files found by grep. Can’t get it to work

Related Question