Shell – Use paste with row-major input

pasteshell-scripttext processing

I can create a file with multiple columns from a single-column input via paste like this:

some_command | paste - -

This works when some_command produces the data in column-major format. In other words, the input

results in the output

1   2
3   4

However, I want the opposite, i.e. I want

1   3
2   4

Background: I want to collect all N’th columns from M files into one aggregate file. I tried doing this via:

cut -f 5 "${files[@]}" | paste - - - - - …

(with M -s). But as mentioned before, this doesn’t work, as paste expects column-major input. I can’t help but think that there should be a coreutils (or pure Bash) solution for this.

Best Answer

If I correctly understand the question, you could try with pr:

cut -f 5 "${files[@]}" | pr -5 -s' ' -t -l 40

where -5 is the number of columns, -s' ' is the separator (space) and -l 40 is the page length (40 lines).

Without coreutils, one could use split to create pieces of N lines:

split -lN infile

some_command | split -lN

and then paste them together:

paste x* > outfile
rm x*

Related Solutions

Text Processing – Sort but Keep Header Line at the Top

Stealing Andy's idea and making it a function so it's easier to use:

# print the header (the first line of input)
# and then run the specified command on the body (the rest of the input)
# use it in a pipeline, e.g. ps | body grep somepattern
body() {
    IFS= read -r header
    printf '%s\n' "$header"
    "$@"
}

Now I can do:

$ ps -o pid,comm | body sort -k2
  PID COMMAND
24759 bash
31276 bash
31032 less
31177 less
31020 man
31167 man
...

$ ps -o pid,comm | body grep less
  PID COMMAND
31032 less
31177 less

Lum – Select certain column of each file, paste to a new file

with paste under bash you can do:

paste <(cut -f 4 1.txt) <(cut -f 4 2.txt) .... <(cut -f 4 20.txt)

With a python script and any number of files (python scriptname.py column_nr file1 file2 ... filen):

#! /usr/bin/env python

# invoke with column nr to extract as first parameter followed by
# filenames. The files should all have the same number of rows

import sys

col = int(sys.argv[1])
res = {}

for file_name in sys.argv[2:]:
    for line_nr, line in enumerate(open(file_name)):
        res.setdefault(line_nr, []).append(line.strip().split('\t')[col-1])

for line_nr in sorted(res):
    print '\t'.join(res[line_nr])

Best Answer

Related Solutions

Text Processing – Sort but Keep Header Line at the Top

Lum – Select certain column of each file, paste to a new file

Related Question