How to Randomly Sample a Subset of a File – Command Line Guide

Sort of. The shell has no idea what the commands you are running will do, it just connects the output of one to the input of the other.

If grep finds more than 10 lines that say "hello world" then head will have all 10 lines it wants, and close the pipe. This will cause grep to be killed with a SIGPIPE, so it does not need to continue scanning a very large file.

The reason there isn't a 'standard tool' is because the logic is - as you've found - quite simple. The limiting factor is that you must do a deep directory traversal, and that's always an expensive process.

It doesn't really matter what approach you take in terms of scripting tools - the 'cost' is the disk IO.

So the optimisations I'd suggest would be:

• Don't walk the whole FS. Bail out of your traversal when you've found enough. (find | shuf and find | head won't do this).
• You can probably approximate directory sizing by referring to last traversals, and 'skip ahead' by some margin.
• statting files as you go and recording mtime will help you build both lists. If you generate a random number, and select the 'last' file, and the last recent file before that number.

Something like this (In perl but I'm sure you could do it in Python)

#!/usr/bin/env perl
use strict;
use warnings;

use File::Find;

my $random_file;
my $recent_random_file;

my $limit = rand ( 7_000_000 ); #ideally set to file count on fs. 

sub search {
    if ( $count++ > $limit ) { 
        $File::Find::prune = 1; #stop traversing
        return; 
    }
    return unless -f; 
    if ( -M $File::Find::name < 1 ) { $recent_random_file = $File::Find::name }; 
    $random_file = $File::Find::name; 
}

find ( \&search, "/path/to/search");
print "$recent_random_file $random_file\n";