Transform tar into cpio without rooting

archivecpionot-root-usertar

I have a tarball containing many files with root:root owner and/or special device inodes. cpio only accepts paths existing in the file system when it is creating a cpio archive. And I don't want to have to sudo in order to transform tar into cpio, while I want to preserve all the permissions, owners, and special inodes.

Is there some clean way of handling this?

EDIT1:

I believe that fakeroot can be seen as somewhat clean way. However it does not scale as expected – nearly 1,000x speed difference:

[user@computer root]$ ls -l ../fanbox.tar
-rw-rw-r-- 1 user user 56555520 May 22 03:33 ../fanbox.tar
[user@computer root]$ time tar -x --delay-directory-restore -f ../fanbox.tar
tar: dev/null: Cannot mknod: Operation not permitted
tar: dev/random: Cannot mknod: Operation not permitted
tar: dev/urandom: Cannot mknod: Operation not permitted
tar: Exiting with failure status due to previous errors

real    0m0.255s
user    0m0.062s
sys 0m0.193s
[user@computer root]$ rm -rf *
[user@computer root]$ time fakeroot tar -x --delay-directory-restore -f ../fanbox.tar

real    3m49.381s
user    0m0.812s
sys 0m2.760s
[user@computer root]$

Based on the output of time command I guess it is because of the communication between fakeroot and faked.

For reference, there is no much difference between a 2M tarball and 50M tarball when I changed fakeroot into sudo bash in my script. And also I believe that the problem is the number of files in the tarball, not the size: I used the same script on a ~10M tarball with two 5M binaries, and the script is not so slow.

Best Answer

You can use fakeroot. As the name says, it fakes the root user, by intercepting serveral syscalls with a LD_LIBRARY_PATH/LD_PRELOAD library wrapper, to have the process believe it's running as root. This has been created for the purpose of building and packaging applications without having to be root, including using make install which typically would be run as root. It's especially well suited to create archives.

During this, a forked daemon faked will run in order to remember all faked file ownership rights or informations on special files the child processes believes they made. So all operations have to be done in the same "instance", or faked will exit and forget what it was remembering.

$ fakeroot
# tar xf ...
# find foo ... | cpio -o ...
# exit
$

Other example showing faked's interaction:

$ mknod /tmp/vaporware b 8 0
mknod: /tmp/vaporware: Operation not permitted
$ fakeroot
# mknod /tmp/vaporware b 8 0
# ls -l /tmp/vaporware
brw-r--r-- 1 root root 8, 0 May 18 08:33 /tmp/vaporware
# exit
$ ls -l /tmp/vaporware
-rw-r--r--. 1 user user 0 May 18 08:33 /tmp/vaporware

Related Solutions

Tar – Transform a Tar Archive’s Paths Without Extracting It

You could mount the archive with archivemount or mountavfs and recreate it again

archivemount tarfile.tar /mnt
cd /mnt
tar cf /tmp/tarfile.tar --transform 's/foo/bar/' .

write operations on the archive filesystem will perfom a full rewrite on umount, so don't seem a good option for large files.

EDIT

I don't know implementation details but seem like we are saving the write files to filesystem step.

Just test to solve dudes, (over a tar of my /usr)

#!/bin/bash

# try to avoid slab cache issues 
cat /tmp/usr.tar > /dev/null

T="$(date +%s)"
tar xf /tmp/usr.tar
tar cf usr.tar usr --transform 's/usr/foo/'
T="$(($(date +%s)-T))"
echo "Tar/Untar seconds: ${T}"

T="$(date +%s)"
archivemount -o readonly -o nobackup /tmp/usr.tar /mnt
tar cf usr.tar /mnt  --transform 's/usr/foo/'
umount /mnt
T="$(($(date +%s)-T))"
echo "Archivemount seconds: ${T}"

T="$(date +%s)"
mountavfs
cd '/root/.avfs/tmp/usr.tar#'
tar cf /tmp/test/usr.tar   --transform 's/usr/foo/' .
T="$(($(date +%s)-T))"
echo "Avfs seconds: ${T}"

Output:

Tar/Untar seconds: 480
Archivemount seconds:  failure, a lot of read errors.
Avfs seconds: 217

So Avfs wins!.

Cpio VS tar – what the best archive solution in order to compress hundred of directories to one file

cpio (the older of the two utilities counting shipping with UNIX) only used to have hard link support for the -p option (i.e. copying from filesystem to filesystem), but the newc output format (not the default one cpio uses) also supports hard links in the output file. (GNU) tar supports hard links without any special options. A comparison can be found here.

So if you run a test with a large hard linked file and 100 small files:

$ mkdir tmp
$ dd if=/dev/urandom of=tmp/blabla bs=1k count=1024
1024+0 records in
1024+0 records out
1048576 bytes (1,0 MB) copied, 0,0764345 s, 13,7 MB/s
$ ln tmp/blabla tmp/hardlink
$ tar cvf tmp.tar tmp
$ find tmp -print0 | cpio -0o > out.cpio
4104 blocks
$ find tmp -print0 | cpio -0o --format=newc > outnewc.cpio
2074 blocks
$ xz -9k out.tar outnewc.cpio
$ bzip2 -9k out.tar outnewc.cpio
$ ls -l out*
-rw-rw-r-- 1 anthon users 2101248 Nov 23 12:30 out.cpio
-rw-rw-r-- 1 anthon users 1061888 Nov 23 12:30 outnewc.cpio
-rw-rw-r-- 1 anthon users 1055935 Nov 23 12:30 outnewc.cpio.bz2
-rw-rw-r-- 1 anthon users 1050652 Nov 23 12:30 outnewc.cpio.xz
-rw-rw-r-- 1 anthon users 1157120 Nov 23 12:30 out.tar
-rw-rw-r-- 1 anthon users 1055402 Nov 23 12:30 out.tar.bz2
-rw-rw-r-- 1 anthon users 1050928 Nov 23 12:30 out.tar.xz

You see that the uncompressed versions (outnewc.cpio and out.tar) give cpio an advantage and that compressing them with xz -9 gives better results than bzip2 -9 (gzip is usually much worse than either). And that compression with xz minimizes the tar and cpio output difference. Compression is however heavily dependent on the data, and also on the ordering of the data in the archives, so you should really test this on (a sample of) your real data.

If you want to compress in parallel, you might want to look at my article here

Best Answer

Related Solutions

Tar – Transform a Tar Archive’s Paths Without Extracting It

Cpio VS tar – what the best archive solution in order to compress hundred of directories to one file

Related Question