Bash Find Command – Report Total Size with ls

bashfindls

I am trying to get the total size of files satisfying a find, e.g.:

ls  $(find -maxdepth 2 -type f)

However, this kind of invocation of ls does not produce the total size as well.

Best Answer

Believe it or not you can do this with find and du. I used a similar technique that I wrote up on my blog a while a go. That article is titled: [one-liner]: Calculating Disk Space Usage for a List of Files Using du under Linux.

The gist of that post is a command such as this:

$ find -maxdepth 2 -type f | tr '\n' '\0' | du -ch --files0-from=-

Example

This will list the size of all the files along with a summary total.

$ find -maxdepth 2 -type f | tr '\n' '\0' | du -ch --files0-from=- | tail -10
0   ./92086/2.txt
0   ./92086/5.txt
0   ./92086/14.txt
0   ./92086/19.txt
0   ./92086/18.txt
0   ./92086/17.txt
4.0K    ./load.bash
4.0K    ./100855/plain.txt
4.0K    ./100855/tst_ccmds.bash
21M total

NOTE: This solution requires that du support the --files0-from= switch which is a GNU switch, to my knowledge.

excerpt from du man page

--files0-from=F
          summarize disk usage of the NUL-terminated file names specified in 
          file F; If F is - then read names from standard input

Also this method suffers from not being able to deal with special characters in file names, such as spaces and non-printables.

Examples

du: cannot access `./101415/fileD': No such file or directory
du: cannot access `E': No such file or directory

These could be dealt with by introducing more tr .. .. commands to substitute them with alternative characters. However there is a better way, if you have access to GNU's find.

Improvements

If your version of find offers the --print0 switch then you can use this incantation which deals with files that have spaces and/or special characters that aren't printable.

$ find -maxdepth 2 -type f -print0 | du -ch --files0-from=- | tail -10
0   ./92086/2.txt
0   ./92086/5.txt
0   ./92086/14.txt
0   ./92086/19.txt
0   ./92086/18.txt
0   ./92086/17.txt
4.0K    ./load.bash
4.0K    ./100855/plain.txt
4.0K    ./100855/tst_ccmds.bash
21M total

Related Solutions

Find Command – How to Count Text Occurrences in Files

For question 1, you can do this:

find . -name "*.txt" | xargs grep -i "abc" | wc -l

This counts the total number of matches for abc in all text files.

And for question 2, I came up with:

find . -name "*.txt" -exec grep -i "abc" {} + | cut -d: -f1 | sort | uniq | wc -l

This gets just the unique filenames from the list of matches and counts them (the sort is probably not needed).

As pointed out by miracle173, grep comes with a "one match per file" flag so the command can be shortened to just:

find . -name "*.txt" -exec grep -il "abc" {} + | wc -l

Shell – Find, count and sort all audio files. ALAC (M4A) files

You should decompose your goal into several steps easier to solve. This will have two advantages:

It will be easier to solve,
The resulting code will be clearer and more reusable.

The scripts below basically follows these steps:

Generate raw statistic files. An easy way is to append the file size and the file name in a temporary file named after the original file extension. So, if you have the file /path/to/foo.mp3 which is 3000000 large, it will append 3000000 /path/to/foo.mp3 at the end of a temporary file named mp3.
Handle specific cases. Here it will process the temporary file m4a and create the two other files m4a_aac and m4a_alac based on the test you gave in the question.
Generate output. All the needed information being now available, it just has to:
- Count the number of line in each temporary file to determine the number of file of this type,
- Sum up each size to get the total size of files of this type.

Here is the script:

#!/bin/sh

# This script takes the searched directory as first parameter.
# For instance: ./this-script.sh ~/Music

: ${1:?"You must pass the search directory as first parameter."}
searchdir="$1"

# Create a temporary directory
statsdir=""
trap 'rm -rf $statsdir' EXIT
statsdir=$(mktemp -d "/tmp/tmp.XXXXXXXXXX") || exit 1

# Generate one listing file per extension
awkscript='/\.[[:alnum:]]+$/ {print $0 >statsdir"/"$(NF)}'
# For Linux: stat -c "%s %n"
# For Mac: stat -f "%z %N"
find "$searchdir" -type f -exec stat -f "%z %N" {} + | \
    awk -F '.' -v statsdir="$statsdir" "$awkscript"

# Distinguish between m4a/AAC and m4a/ALAC
if [ -f "$statsdir/m4a" ]; then
    input="$statsdir/m4a"
    while IFS= read -r line; do
        filename=${line#* }
        if avprobe "$filename" 2>&1 | grep -q 'Audio: alac'; then
            echo "$line" >> "$statsdir/m4a_alac"
        else
            echo "$line" >> "$statsdir/m4a_aac"
        fi
    done < "$input"
    rm "$statsdir/m4a"
fi

# Generate and display result
{
    printf "Type Count Size\n"
    for extension in $(ls "$statsdir"); do
        count=$(wc -l "$statsdir/$extension" | cut -d ' ' -f 1)
        totalsize=$(awk '{s+=$1} END {print s}' "$statsdir/$extension")
        printf "%s %d %d\n" "$extension" "$count" "$totalsize"
    done
} | column -t