Extract random sample of N lines based on pattern

awkrandomsedtext processing

I have a file formatted like this:

train/t/temple/east_asia/00000025.jpg 94
train/t/temple/east_asia/00000865.jpg 94
...
train/s/swamp/00000560.jpg 92
train/s/swamp/00000935.jpg 92
....
train/m/mountain/00000428.jpg 68
train/m/mountain/00000126.jpg 68

The last number is the class number. I have 50 different classes, and each class has 1,000 lines. I would like to take a random sample of size N from each class, and store the result in another text file.

Best Answer

Since your lines are grouped by class, you could (with gnu tools)split the file into pieces and use the --fiter option to pipe each piece to shuf to extract N random lines from it:

split --filter='shuf -n N' infile > outfile

Note that split defaults to 1000 lines - which is what you need in this particular case. If the requirements change you'll have to pass the number of lines via -l
e.g. to split into pieces of 200 lines and extract 30 random lines from each piece:

split -l 200 --filter='shuf -n 30' infile > outfile

Related Solutions

Shell – Print random lines respecting the order of the source file

Use this:

nl file | shuf -n2 | sort -n | cut -f2-

nl to number the lines,
shuf to shuffle and limit the output to 2 lines (-n),
sort to rebuild the original order,
and cut to remove the numeration of nl.

It will print 2 lines of your file in the original order of the file. Use shuf -n X, where X can be any number.

How to edit the last n lines in a file

To replace commas with semicolons on the last n lines with ed:

n=3
ed -s input <<< '$-'$((n-1))$',$s/,/;/g\nwq'

Splitting that apart:

ed -s = run ed silently (don't report the bytes written at the end)
'$-' = from the end of the file ($) minus ...
$((n-1)) = n-1 lines ...
( $' ... ' = quote the rest of the command to protect it from the shell )
,$s/,/;/g = ... until the end of the file (,$), search and replace all commas with semicolons.
\nwq = end the previous command, then save and quit

To replace commas with semicolons on the last n lines with sed:

n=3
sed -i "$(( $(wc -l < input) - n + 1)),\$s/,/;/g" input

Breaking that apart:

-i = edit the file "in-place"
$(( ... )) = do some math:
$( wc -l < input) = get the number of lines in the file
-n + 1 = go backwards n-1 lines
,\$ = from n-1 lines until the end of the file:
s/,/;/g = replace the commas with semicolons.

Best Answer

Related Solutions

Shell – Print random lines respecting the order of the source file

How to edit the last n lines in a file

Related Question