Hashsum Calculation – How to Simultaneously Calculate Multiple Digests (md5, sha256)

hashsumparallelismshell-script

Under the assumption that disk I/O and free RAM is a bottleneck (while CPU time is not the limitation), does a tool exist that can calculate multiple message digests at once?

I am particularly interested in calculating the MD-5 and SHA-256 digests of large files (size in gigabytes), preferably in parallel. I have tried openssl dgst -sha256 -md5, but it only calculates the hash using one algorithm.

Pseudo-code for the expected behavior:

for each block:
    for each algorithm:
        hash_state[algorithm].update(block)
for each algorithm:
    print algorithm, hash_state[algorithm].final_hash()

Best Answer

Check out pee ("tee standard input to pipes") from moreutils. This is basically equivalent to Marco's tee command, but a little simpler to type.

$ echo foo | pee md5sum sha256sum
d3b07384d113edec49eaa6238ad5ff00  -
b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c  -

$ pee md5sum sha256sum <foo.iso
f109ffd6612e36e0fc1597eda65e9cf0  -
469a38cb785f8d47a0f85f968feff0be1d6f9398e353496ff7aa9055725bc63e  -

Related Solutions

How to rename multiple files to their contents’ MD5 sum

Glenn's answer is good; here's a refinement for multiple files:

md5sum file1 file2 file3 | # or *.txt, or whatever
    while read -r sum filename; do
        mv -v "$filename" "$sum"
    done

If you're generating files with find or similar, you can replace the md5sum invocation with something like find . <options> -print0 | xargs -0 md5sum (with the output also piped into the shell loop).

This is taking the output of md5sum, which consists of multiple lines with a sum and then the file it corresponds to, and piping it into a shell loop which reads each line and issues a mv command that renames the file from the original name to the sum. Any files with identical sums will be overwritten; however, barring unusual circumstances (like if you're playing around with md5 hash collisions), that will mean they had the same contents, so you don't lose any data anyway. If you need to introduce other operations on each file, you can put them in the loop, referring to the variables $filename and $sum, which contain the original filename and the MD5 sum respectively.

Shell – How to use GNU parallel to calculate sha256 hash

Using xargs (and assuming that you have an implementation of this utility that supports -0 and -P):

printf '%s\0' * | xargs -0 -L 1 -P 16 sh -c 'sha256sum "$1" > "$1".sha' sh

This would pass all names in the current directory as a nul-terminated list to xargs. The xargs utility would call an in-line sh script for each one of these names, starting at most 16 concurrent processes. The in-line script takes the argument and runs sha256sum on it, outputting the result to a file of a similar name.

Note that this would also possibly pick up .sha files created in a previous run of the same pipeline. To avoid this, use a slightly more sophisticated glob than * to match the particular names that you'd want to process. For example, in bash:

shopt -s extglob
printf '%s\0' !(*.sha) | xargs ...as above...

Note also that running sha256sum on large files in parallel is likely to be disk bound rather than CPU bound and that you may possibly see similar speed of operation with a smaller number of parallel tasks.

For a GNU parallel equivalent, replace xargs with parallel.

In the zsh shell, you can do it like

autoload -U zargs
setopt EXTENDED_GLOB

zargs -P 16 -L 1 -- (^(*.sha)) -- sh -c 'sha256sum "$1" > "$1".sha' sh

Best Answer

Related Solutions

How to rename multiple files to their contents’ MD5 sum

Shell – How to use GNU parallel to calculate sha256 hash

Related Question