Gzip, send to server, ungzip

gzipscriptingtar

I am working on a script to backup and gzip a set of MySQL tables, scp/ssh them to a different server and then unpack them
current script is:

#!/bin/bash

DATE=`date +"%d_%b_%Y_%H%M"`
BACKUP_DIR=/mnt/data/backups/saturday/

echo Creating new directory /mnt/data/backups/saturday/fathom_$DATE
sudo mkdir /mnt/data/backups/saturday/fathom_$DATE
sudo chmod 777 /mnt/data/backups/saturday/fathom_$DATE

mysqldump pacific_fathom user_types > /mnt/data/backups/saturday/fathom_$DATE/user_types.sql
echo Dumping users ...
mysqldump pacific_fathom users > /mnt/data/backups/saturday/fathom_$DATE/users.sql
echo Dumping users_roles ...
mysqldump pacific_fathom users_roles > /mnt/data/backups/saturday/fathom_$DATE/users_roles.sql

tar -zvcpf $BACKUP_DIR/PlatformDB-$DATE.tar.gz /mnt/data/backups/saturday/fathom_* | ssh [email protected] 'tar -xzf - -C /mnt/data/backups/saturday'

echo Finished!

the backup works and it will zip the files but it tells me it errors with

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

The output file it creates is PlatformDB-22_Jul_2021_1553.tar.gz

It never moves to the ssh part (checked last in the remote server and there are no logins there) so I am a bit confused as to why I am get this error. The tar.gz file it creates can be unpacked using the tar -xzf - -C /mnt/data/backups/saturday part of the script. I do have to replace the - with the file name. I do not think that would make a difference in the script though. Also that would not work in an automated environment. Any help would be GREATLY appreaciated!!!

Best Answer

The command tar -zvcpf ...$DATE.tar.gz /mnt/data/backups/saturday/fathom_* writes the gzipped backup to $DATE.tar.gz and outputs the list of files that are backed up.

This file list is then piped into ssh [email protected] 'tar -xzf - .... Obviously, the file list is not in gzipped format, which causes the error.

Solution: Send the content of $DATE.tar.gz to the ssh command. For example first create the local backup file, then cat $DATE.tar.gz | ssh .... Or, if you don't need the local backup file, create the backup on standard output:

tar -zvcpf - /mnt/data/backups/saturday/fathom_* | ssh ...

You can also try the tee command and write the backup to both standard output and a local file:

tar -zvcpf - /mnt/data/backups/saturday/fathom_* | tee $DATE.tar.gz | ssh ...

As a side remark not really relevant to this question, I just noticed that tar writes the file list to stdout when the archive is written to a file, and to stderr when the archive is written to stdout. Smart!

gzip -l -v

gzip-compressed files contain already a hash (not secure though, see this SO post):

$ echo something > foo
$ gzip foo
$ gzip -v -l foo.gz 
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 18b1f736 Feb  8 22:34                  34                  10 -20.0% foo

One can combine the CRC and uncompressed size to get a quick fingerprint:

gzip -v -l foo.gz | awk '{print $2, $7}'

cmp

For checking whether two bytes are equal or not, use cmp file1 file2. Now, a gzipped file has some header with the data and footer (CRC plus original size) appended. The description of the gzip format shows that the header contains the time when the file was compressed and that the file name is a nul-terminated string that is appended after the 10-byte header.

So, assuming that the file name is constant and the same command (gzip "$name") is used, one can check whether two files are different by using cmp and skipping the first bytes including the time:

cmp -i 8 file1 file2

Note: the assumption that the same compression options is important, otherwise the command will always report the file as different. This happens because the compression options are stored in the header and may affect the compressed data. cmp just looks at raw bytes and do not interpret it as gzip.

If you have filenames of the same length, then you could try to calculate the bytes to be skipped after reading the filename. When the filenames are of different size, you could run cmp after skipping bytes, like cmp <(cut -b9- file1) <(cut -b10- file2).

zcmp

This is definitely the best way to go, it first compresses data and starts comparing the bytes with cmp (really, this is what is done in the zcmp (zdiff) shellscript).

One note, do not be afraid of the following note in the manual page:

When both files must be uncompressed before comparison, the second is uncompressed to /tmp. In all other cases, zdiff and zcmp use only a pipe.

When you have a sufficiently new Bash, compression will not use a temporary file, just a pipe. Or, as the zdiff source says:

# Reject Solaris 8's buggy /bin/bash 2.03.

Best Answer

Related Solutions

How to check if two gzipped files are equal

gzip -l -v

cmp

zcmp

Related Question