Problem with paste and standard output in linux

io-redirectionpastetext processing

I have two files that I am trying to merge, one file is:

linux$ cat temp2
linear_0_A_B linear_0_B_A
103027.244444 102714.177778
103464.311111 102876.266667
103687.422222 103072.711111
103533.244444 102967.733333
103545.066667 102916.933333
103621.555556 103027.511111
104255.776536 103006.256983
103712.178771 102877.139665
103817.555556 103198.488889
103701.422222 103133.200000

And the other file is:

linux$ cat temp
linear_1_A_B linear_1_B_A
118620.444444 109212.355556
108408.488889 105744.444444
108136.311111 105174.933333
108627.688889 105390.044444
108356.577778 105412.888889
108559.204420 105667.933702
108469.392265 105547.314917
109032.044693 105497.698324
108925.866667 105986.222222
107975.733333 105070.000000

I want to paste the columns in temp into temp2, and retain the temp2 file like this:

linux$ paste temp2 temp
linear_0_A_B linear_0_B_A       linear_1_A_B linear_1_B_A
103027.244444 102714.177778     118620.444444 109212.355556
103464.311111 102876.266667     108408.488889 105744.444444
103687.422222 103072.711111     108136.311111 105174.933333
103533.244444 102967.733333     108627.688889 105390.044444
103545.066667 102916.933333     108356.577778 105412.888889
103621.555556 103027.511111     108559.204420 105667.933702
104255.776536 103006.256983     108469.392265 105547.314917
103712.178771 102877.139665     109032.044693 105497.698324
103817.555556 103198.488889     108925.866667 105986.222222
103701.422222 103133.200000     107975.733333 105070.000000

But when I do standard output, and display temp2, the result is not the same.

linux$ paste temp2 temp > temp2
linux$ cat temp2
        linear_1_A_B linear_1_B_A
        118620.444444 109212.355556
        108408.488889 105744.444444
        108136.311111 105174.933333
        108627.688889 105390.044444
        108356.577778 105412.888889
        108559.204420 105667.933702
        108469.392265 105547.314917
        109032.044693 105497.698324
        108925.866667 105986.222222
        107975.733333 105070.000000

How to resolve??

Best Answer

You wrote in your last block,

linux$ paste temp2 temp > temp2

You cannot do this. (Well you can, but it won't work.) What happens here is that the shell truncates temp2 ready to send output from the paste command. The paste temp2 temp command then runs - but by this stage temp2 is already zero length.

What you can do instead is this, which uses a third file to collect the output and then replaces your temp2 with its content. The && ensures that the content is only replaced if the paste "succeeded", and the rm -f removes the transient temp3 file if the mv wasn't triggered, or failed in some unexpected way.

paste temp2 temp > temp3 && mv -f temp3 temp2
rm -f temp3

Related Solutions

UNIX paste columns and insert zeros for all missing values

If column-order is important, i.e. numbers from the same file should be kept in the same column, you need to add padding while reading the different files. Here is one way that works with GNU awk:

merge.awk

# Set k to be a shorthand for the key
{ k = $1 SUBSEP $2 }

# First element with this key, add zeros to align it with other rows
!(k in h) {
  for(i=1; i<=ARGIND-1; i++)
    h[k] = h[k] OFS 0 
}

# Remember the data element
{ h[k] = h[k] OFS $3 }

# Before moving to the next file, ensure that all rows are aligned
ENDFILE {
  for(k in h) {
    if(split(h[k], a) < ARGIND)
      h[k] = h[k] OFS 0
  }
}

# Print out the collected data
END {
  for(k in h) {
    split(k, a, SUBSEP)
    print a[1], a[2], h[k]
  }
}

Here are some test files: f1, f2, f3 and f4:

$ tail -n+1 f[1-4]
==> f1 <==
xyz desc1 21
uvw desc2 22
pqr desc3 23

==> f2 <==
xyz desc1 56
uvw desc2 57

==> f3 <==
xyz desc1 87
uvw desc2 88

==> f4 <==
xyz desc1 11
uvw desc2 12
pqr desc3 13
stw desc1 14
arg desc2 15

Test 1

awk -f merge.awk f[1-4] | column -t

Output:

pqr  desc3  23  0   0   13
uvw  desc2  22  57  88  12
stw  desc1  0   0   0   14
arg  desc2  0   0   0   15
xyz  desc1  21  56  87  11

Test 2

awk -f merge.awk f2 f3 f4 f1 | column -t

Output:

pqr  desc3  0   0   13  23
uvw  desc2  57  88  12  22
stw  desc1  0   0   14  0
arg  desc2  0   0   15  0
xyz  desc1  56  87  11  21

Edit:

If the output should be tab-separated, set the output field separator accordingly:

awk -f merge.awk OFS='\t' f[1-4]

Shell – Use paste with row-major input

If I correctly understand the question, you could try with pr:

cut -f 5 "${files[@]}" | pr -5 -s' ' -t -l 40

where -5 is the number of columns, -s' ' is the separator (space) and -l 40 is the page length (40 lines).

Without coreutils, one could use split to create pieces of N lines:

split -lN infile

some_command | split -lN

and then paste them together:

paste x* > outfile
rm x*