Shell – easiest way to rearrange columns and manipulate text file

shell-scripttext processing

Learning linux in school and working on manipulating text files at the moment. Looking to learn a few shortcuts here and there along the way.
Currently I have a text file with content such as:

First    Last   111 E. Road    New York    NY
First2   Last2  222 w. Road    Newark      NJ

We're supposed to write a script to rearrange the columns and comma delimit instead of tab delimit. What i did was simply cut each field and put into its own tmpfile and then pasted together as such:

paste tmplast tmpfirst tmpstate tmpaddress | tr '\t' ',' > finished

Is there a faster way rather than cutting everything into a tmp file and pasting together? I'm very new to linux and the only commands I've learned for manipulating files are like tr and sed.

Best Answer

You can do it in awk like this:

awk 'BEGIN {FS="\t"; OFS=","} {print $2, $1, $5, $3, $4}' file

FS and OFS specify the "(input) field separator" and "output field separator", and then the order in which to print fields can be specified explicitly using the $ notation. (No temporary files needed.)

Output:

Last,First,NY,111 E. Road,New York
Last2,First2,NJ,222 w. Road,Newark

Related Solutions

Rearrange text file by prepending data rows with header row

You can use awk for this:

awk '/^2009/{a=$0;next}{print a" "$0} ' file.txt

This will only prepend a space to any lines before the first match of 2009, you can set a default string to prepend like this:

awk 'BEGIN{a="My default prepend string";}/^2009/{a=$0;next}{print a" "$0} ' file.txt

Fold and text columns

GNU fold and GNU fmt only understand bytes, not characters. To wrap to a certain number of characters, you can use sed.

sed 's/.\{20\}/&\n/g' <in.txt
【財經中心、政治中心╱台北報導】看不慣政
府施政效率緩慢，鴻海集團董事長郭台銘動念
選總統！《壹週刊》報導，在川普勝選當晚，
郭召集鴻海高層幹部，進行美國總統大選換人
後的應變策略演練，讓人驚訝的是，郭詢問在
場幹

If you wanted to break at whitespace (useful for many languages), here's a quick-and-dirty awk script.

awk '
    BEGIN {width = 20}
    NF == 0 {column = 0; print}
    {
        split($0, a);
        for (i in a) {
            w = length(a[i]) + 1;
            column += w;
            if (column > width) {column = w; print ""};
            if (column != w) printf " ";
            printf "%s", a[i];
        }
    }
    END {if (column) print ""}'

In any case make sure that your locale settings are correct. Specifically, LC_CTYPE must designate the right character encoding, e.g. LC_CTYPE=en_US.utf8 or LC_CTYPE=zh_CN.utf8 (any language code that's available on your system will do) for Unicode encoded as UTF-8.

Note that this counts characters, not screen width. Even fixed-width fonts can have double-width characters and this is typically done for Chinese characters, so e.g. a character width of 20 for the text above occupies 40 columns on typical terminals.

Best Answer

Related Solutions

Rearrange text file by prepending data rows with header row

Fold and text columns

Related Question