Shell – /bin/cat: Argument list too long

catlinuxshell-script

I have 119766 files in a folder. They are CSV files. I want to find out total number of lines of all files.

I'm trying to run following command: 

cat * |wc -l

But the following error occurrs:

-bash: /bin/cat: Argument list too long

How can I do that? Is there any way around this?

One thing I would like to add that total number of lines would be very large.

Best Answer

If you want a line-count for each individual file:

find . -type f -exec wc -l {} + | awk '! /^[ 0-9]+[[:space:]]+total$/'

I've excluded the total lines because there will be several of them with this many files being processed. The find ... -exec ... + will try to fit as many filenames onto a single command line as possible, but that will be a LOT less than 119766 files....probably only several thousand (at most) per invocation of wc, and each one will result in its own independent 'total' line.

If you want the total number of lines in all files combined, here's one way of doing it:

find . -type f -exec wc -l {} + | 
    awk '/^[ 0-9]+[[:space:]]+total$/ {print $1}' | 
    xargs | sed -e 's/ /+/g' | bc

This prints only the line counts on the total lines, pipes that into xargs to get the counts all on one line, then sed to transform the spaces into + signs, and then pipes the lot into bc to do the calculation.

Example output:

$ cd /usr/share/doc
$ find . -type f -exec wc -l {} + | 
    awk '/^[ 0-9]+[[:space:]]+total$/ {print $1}' | 
    xargs | sed -e 's/ /+/g' | bc 
53358931
Related Question