How to extract parts of file into separate files in linux using awk or grep or sed commands

awkgrep

I have text file as shown below:

Input file1.txt

.......................
cha21   1       3       5       nar
cha21   2       3       6       piy
cha23   2       3       5       ram
cha23   3       3       3       dam
cha27   5       3       7       pam
................................

And I would like to extract all the 5 columns corresponding to rows of "cha21", "cha23" and "cha27" into 3 different output files(for instance, cha21.txt, cha23.txt, cha27.txt).

output files

cha21.txt

..........
cha21   1       3       5       nar
cha21   2       3       6       piy
......................

cha23.txt

cha23   2       3       5       ram
cha23   3       3       3       dam
...................................

cha27.txt

cha27   5       3       7       pam
...........................

I can do this using grep command 3 times for 3 files…
Is there anyway I can do this all at once, i.e. one command because I need to extract 100 output files.

Best Answer

I would try

awk '{print >> $1 ".txt" ;}'

where

print print the whole line.
>> $1 ."txt" write (append) this line to the file indicated by $1, with .txt added.

edit:

in case you have comment lines, lines with dots, etc

try

awk '$1 ~ /cha/ {print >> $1 ".txt" ;}'

which would only fill file begining with 'cha'.

Related Solutions

UNIX paste columns and insert zeros for all missing values

If column-order is important, i.e. numbers from the same file should be kept in the same column, you need to add padding while reading the different files. Here is one way that works with GNU awk:

merge.awk

# Set k to be a shorthand for the key
{ k = $1 SUBSEP $2 }

# First element with this key, add zeros to align it with other rows
!(k in h) {
  for(i=1; i<=ARGIND-1; i++)
    h[k] = h[k] OFS 0 
}

# Remember the data element
{ h[k] = h[k] OFS $3 }

# Before moving to the next file, ensure that all rows are aligned
ENDFILE {
  for(k in h) {
    if(split(h[k], a) < ARGIND)
      h[k] = h[k] OFS 0
  }
}

# Print out the collected data
END {
  for(k in h) {
    split(k, a, SUBSEP)
    print a[1], a[2], h[k]
  }
}

Here are some test files: f1, f2, f3 and f4:

$ tail -n+1 f[1-4]
==> f1 <==
xyz desc1 21
uvw desc2 22
pqr desc3 23

==> f2 <==
xyz desc1 56
uvw desc2 57

==> f3 <==
xyz desc1 87
uvw desc2 88

==> f4 <==
xyz desc1 11
uvw desc2 12
pqr desc3 13
stw desc1 14
arg desc2 15

Test 1

awk -f merge.awk f[1-4] | column -t

Output:

pqr  desc3  23  0   0   13
uvw  desc2  22  57  88  12
stw  desc1  0   0   0   14
arg  desc2  0   0   0   15
xyz  desc1  21  56  87  11

Test 2

awk -f merge.awk f2 f3 f4 f1 | column -t

Output:

pqr  desc3  0   0   13  23
uvw  desc2  57  88  12  22
stw  desc1  0   0   14  0
arg  desc2  0   0   15  0
xyz  desc1  56  87  11  21

Edit:

If the output should be tab-separated, set the output field separator accordingly:

awk -f merge.awk OFS='\t' f[1-4]

awk grep – Extract Lines Containing Terms from One File to Another Using Grep/Awk

To extract the lines from data.txt with the genes listed in genelist.txt:

grep -w -F -f genelist.txt data.txt > newdata.txt

grep options used:

-w tells grep to match whole words only (i.e. so ABC123 won't also match ABC1234).
-F search for fixed strings (plain text) rather than regular expressions
-f genelist.txt read search patterns from the file

If you want the header (Sample 1, Sample 2, etc) line as well:

grep -w -F -f genelist.txt -e Sample data.txt > newdata.txt

-e Sample also search for "Sample"

To find lines in genelist.txt that aren't in newdata.txt:

grep -v -w -F -f <(sed -E -e 's/(\t|  +).*//' newdata.txt) genelist.txt

-v invert the search, print non-matching lines.

The rest of the grep options are the same, but instead of using a file with the -f option, it's using something called Process Substitution (See also), which allows you to use a command in place of an actual file. Whatever output the command creates is treated as the "file"'s contents.

In this case, we're using the command sed -E -e 's/(\t| +).*//' newdata.txt, which outputs each line of newdata.txt after first deleting everything from either the first TAB character or the first pair of spaces it sees. In other words, the first field (e.g. "Gene A"). I had to use TAB or double space because a) i wasn't sure if your data was space-separated or TAB separated and b) the first fields in your example contained spaces.

sed options used:

-E use extended regular expressions, so we can use plain (, ), and + which are more readable than having to escape them with \ as $, $, \+.
-e 's/(\t| +).*//' specifies the sed script to apply against the input (newdata.txt)

Running that command on your sample data.txt would produce the following output:

$ sed -E -e 's/(\t|  +).*//' data.txt

Gene A
Gene B
Gene C
Gene D

Anyway, the output of that sed command is used as the list of search patterns by the grep command.