Linux – How to compress a large file into smaller parts

compressiongziplinuxtar

I'm looking for a way to compress a large file (~10GB) into several files that wont exceed 150MB each.

Any thoughts?

Best Answer

Compress single file

This will compress file /path/to/your/large/file and creates many files with the prefix compressed.gz in the current directory, each file with a maximum size of 150000000 bytes:

gzip -c /path/to/your/large/file | split -b 150000000 - compressed.gz

Uncompress single file

To uncompress the file resulting in the uncompressed file "/path/to/decrompressed/file" compressed using the command above use:

cat compressed.gz* | zcat > /path/to/decrompressed/file

Related Solutions

Linux – Tar – split files

I don't think tar has functionality built-in to split at an arbitrary size (there is -M for multi-volume, but this relies on the destination media detecting an out-of-space condition), but there are two things you could do:

Generate a single tar.gz file in the usual way, then use the split command to divide it into sections (and cat to join them back together for decompression).
Use dar instead of tar, since this has splitting functionality built in (amongst a lot of other features that tar doesn't have). For example

$ dar -c "$BACKUPNAME" -g "$INPUTFILE.sql" -s300M -z

Linux – How to compress multiple files into one archive using lzma

One thing you could do is use tar:

tar cf files.lzma --lzma file1 file2 ... fileN

Or, simpler,

tar cf files.lzma --lzma file*

That creates the files.lzma archive which you can then unpack using:

tar xf files.lzma