Different count for compressed file listing vs directory listing

compressionlstar

I was taking a backup of a folder, where I used the following command :

sudo tar -zcvf www.tar.gz /var/www/

It successfully compressed, just for verifying the number of files in the compressed archive, and the directory I ran the following commands :

sudo tar -ztvf www.tar.gz | wc -l -> 186364

ls -R /var/www | wc -l -> 196440

How come the count is different, what am I missing ?

Best Answer

Look at the output without piping to wc to see the difference

$ ls -R www | cat
www:
dir1
dir2
file1

www/dir1:
file1

www/dir2:
file1
file2
$ ls -R www | wc -l
11

And

$ tar xvfz www.tar.gz
www/
www/dir1/
www/dir1/file1
www/dir2/
www/dir2/file1
www/dir2/file2
www/file1
$ tar xvfz www.tar.gz |wc -l
7

ls -R just produces some more output.

Related Solutions

On the Fly Compression for a Directory

If read-only access is acceptable, then SquashFS is a good choice.

However, it sounds like you want to be able to do in place updating as well. Btrfs may be an option for you. It is still considered somewhat experimental, but it does support transparent file compression, and is available to try in most distros.

The other approach is to do this in userspace, via FUSE. The most plausible of the options here is probably fusecompress.

Create Archive with tar Including Dotfiles but Excluding Subdirectories

You can use this command to backup all your dotfiles (.<something>) in your $HOME directory:

$ cd ~
$ find . -maxdepth 1 -type f -name ".*" -exec tar zcvf dotfiles.tar.gz {} +

regex using just tar?

method #1

I researched this quite a bit and came up empty. The limiting factor would be that when tar is performing it's excludes, the trailing slash (/) that shows up with directories is not part of the equation when tar is performing its pattern match.

Here's an example:

$ tar -v --create --file=do.tar.gz --auto-compress --no-recursion --exclude={'.','..','.*/'} .*
.qalculate/
.qt/
.qterm/
.qtoctave/
.RapidSVN
.RData
.Rhistory

This variant includes an exclude of .*/ and you can see with the verbose switch turned on to tar, -v, that these directories are passing through that exclude.

method #2

I thought maybe the switches --no-wildcards-match-slash or --wildcards-match-slash would relax the greediness of the .*/ but this had no effect either.

Taking the slash out of the exclude, .* wasn't an option either since that would tell tar to exclude all the dotfiles and dotdirectories:

$ tar -v --create --file=do.tar.gz --auto-compress --no-recursion --exclude={'.','..','.*'} .*
$

method #3 (Ugly!)

So the only other alternative I can conceive is to provide a list of files to tar. Something like this:

$ tar -v --create --file=do.tar.gz --auto-compress --no-recursion --wildcards-match-slash --exclude={'.','..','.*/'} $(ls -aF | grep -E "^\..*[^/]$")
.RapidSVN
.RData
.Rhistory

This approach has issues if the number of files exceeds the maximum amount of space for passing arguments to a command would be one glaring issue. The other is that it's ugly and overly complex.

so what did we learn?

There doesn't appear to be a straight-forward and elegant way to acomplish this using tar & regular expressions. So as to @terdon's comment, find ... | tar ... is really the more appropriate way to do this.