How to wait for execution of parallelized processes and stitch together the outputs

sedsplit

Quite new to doing things on Unix, looking to make a script that does the following things in order:

Take main .tsv file, split into X number of files with Y lines each
Run each split file through a program, which outputs a new .tsv file upon completion
Wait until ALL split files have completed processing, then stitch output files together into one.

I know about using split and sed for splitting files, and I can't imagine getting the split files to run through a Python script is hard either, but the problem is finding out when all executions of the parallelized programs are complete, and THEN stitching their outputs together into one.

With split I know it auto-increments the names and that you can mass parallelize it as seen in this SO question, so I could figure that part out. Is there a way to check for a group of parallelized Python scripts' execution status? How could I accomplish what I'd like to do?

Best Answer

split -l $Y main.tsv main_part_
for part in main_part_*; do
    program $part &
done
wait
echo "all done"

wait is a bash builtin: check the man page for details

Related Solutions

Linux Command-Line – Best Way to Join Files After Splitting

That's just what cat was made for. Since it is one of the oldest GNU tools, I think it's very unlikely that any other tool does that faster/better. And it's not piping - it's only redirecting output.

How to Split a File by Keyword Boundaries

You can use awk for the job:

$ curl -O https://raw.githubusercontent.com/qtproject/qt-mobility\
/d7f10927176b8c3603efaaceb721b00af5e8605b/demos/qmlcontacts/contents/\
example.vcf

$ gawk ' /BEGIN:VCARD/ { close(fn); ++a; fn=sprintf("card_%02d.vcf", a); 
        print "Writing: ", fn } { print $0 > fn; } ' example.vcf
Writing:  card_01.vcf
Writing:  card_02.vcf
Writing:  card_03.vcf
Writing:  card_04.vcf
Writing:  card_05.vcf
Writing:  card_06.vcf
Writing:  card_07.vcf
Writing:  card_08.vcf
Writing:  card_09.vcf

$ cat card_0* > all.vcf
$ cmp example.vcf all.vcf
$ echo $?
0

Details

The awk line works like this: a is counter that is incremented on each BEGIN:VCARD line and at the same time the output filename is constructed using sprintf (stored in fn). For each line the current line ($0) is appended to the current file (named fn).

The last echo $? means that the cmp was successful, i.e. all single files concatenated are equal to the original example vcf example.

Note that the output redirection in awk works differently than in shell. That means that with > fn awk first checks if the file is already open. If it is already open then awk appends to it. If it is not then it opens and truncates it.

Because of this redirection logic we have to explicitly close the implicitly opened files, since otherwise the call would hit the open file limit in cases where the input file contains many records.

Best Answer

Related Solutions

Linux Command-Line – Best Way to Join Files After Splitting

How to Split a File by Keyword Boundaries

Details

Related Question