I need to sort a bed
file randomly 10000 times and take the top 1000 rows each time. Currently, I am using the following code:
for i in {1..100}; do
for j in {1..100}; do
sort -R myfile.bed_sorted | tail -n 1000 > myfile.bed.$i.$j.bed
done
done
It takes almost 6 hours to do this for each file. I have around 150 of them to be worked out. Is there a faster solution for this?
A sample of the data (myfile.bed_sorted) I have:
chr1 111763899 111766405 peak1424 1000 . 3224.030 -1 -1
chr1 144533459 144534584 peak1537 998 . 3219.260 -1 -1
chr8 42149384 42151246 peak30658 998 . 3217.620 -1 -1
chr2 70369299 70370655 peak16886 996 . 3211.600 -1 -1
chr8 11348914 11352994 peak30334 990 . 3194.180 -1 -1
chr21 26828820 26830352 peak19503 988 . 3187.820 -1 -1
chr16 68789901 68791150 peak11894 988 . 3187.360 -1 -1
chr6 11458964 11462245 peak26362 983 . 3169.750 -1 -1
chr1 235113793 235117308 peak2894 982 . 3166.000 -1 -1
chr6 16419968 16422194 peak26522 979 . 3158.520 -1 -1
chr6 315344 321339 peak26159 978 . 3156.320 -1 -1
chr1 111756584 111759633 peak1421 964 . 3110.520 -1 -1
chrX 12995098 12997685 peak33121 961 . 3100.000 -1 -1
chr9 37408601 37410262 peak32066 961 . 3100.000 -1 -1
chr9 132648603 132651523 peak32810 961 . 3100.000 -1 -1
chr8 146103178 146104943 peak31706 961 . 3100.000 -1 -1
chr8 135611963 135614649 peak31592 961 . 3100.000 -1 -1
chr8 128312253 128315935 peak31469 961 . 3100.000 -1 -1
chr8 128221486 128223644 peak31465 961 . 3100.000 -1 -1
chr8 101510621 101514237 peak31185 961 . 3100.000 -1 -1
chr8 101504210 101508005 peak31184 961 . 3100.000 -1 -1
chr7 8173062 8174642 peak28743 961 . 3100.000 -1 -1
chr7 5563424 5570618 peak28669 961 . 3100.000 -1 -1
chr7 55600455 55603724 peak29192 961 . 3100.000 -1 -1
chr7 35767878 35770820 peak28976 961 . 3100.000 -1 -1
chr7 28518260 28519837 peak28923 961 . 3100.000 -1 -1
chr7 104652502 104654747 peak29684 961 . 3100.000 -1 -1
chr6 6586316 6590136 peak26279 961 . 3100.000 -1 -1
chr6 52362185 52364270 peak27366 961 . 3100.000 -1 -1
chr6 407805 413348 peak26180 961 . 3100.000 -1 -1
chr6 32936987 32941352 peak26978 961 . 3100.000 -1 -1
chr6 226477 229964 peak26144 961 . 3100.000 -1 -1
chr6 157017923 157020836 peak28371 961 . 3100.000 -1 -1
chr6 137422769 137425128 peak28064 961 . 3100.000 -1 -1
chr5 149789084 149793727 peak25705 961 . 3100.000 -1 -1
chr5 149778033 149783125 peak25702 961 . 3100.000 -1 -1
chr5 149183766 149185906 peak25695 961 . 3100.000 -1 -1
Best Answer
Assuming you have enough memory to slurp the file, you could try
Since you want to do this 10000 times, I would recommend integrating the repetition into the script and shuffling the indices instead of the array itself to speed things up:
The above created 10000 files of 1000 lines each from a file that contained 37000 rows (your example file repeated 1000 times). As you can see, it took slightly more than three minutes on my system.
Explanation
use List::Util 'shuffle';
: this imports a Perl module that provides theshuffle()
function which randomizes an array.@l=<>;
: load the input file (<>
) into the array@l
.for $i (1..10000){}
: run this 10000 times.@r=shuffle(0..$#l);
:$#l
is the number of elements in@l
so@r
is now a randomized list of the index numbers of the array@l
(the input file's lines).open(my $fh, ">","file.$i.bed");
: open a file calledfile.$i.bed
for writing.$i
will take values from 1 to 10000.print $fh @l[@r[0..999]]
: take the first 1000 indices in the shuffled array and print the corresponding lines (elements of@l
).Another approach is to use
shuf
(thanks @frostschutz):