I've been playing around with obnam these last few days, and although it looks very promising and seems to offer basically everything I ever wanted in a backup tool, I am pretty disappointed in its performance. In fact, it's so slow, I'm suspecting that obnam is not even at fault here, but something in my environment is causing it.
So I'm mainly wondering, if anyone else is using obnam or knows its internals well enough to maybe identify the problem.
From what I could tell so far, obnam seems to fork an individual gpg process for each file that is backed up. Judging from htop, strace, and iostat, the speed of an initial backup is mostly bounded by the constant forking, while the CPU and the drives (no networking is involved) are mostly idling below 20% utilization.
My backup amounts to about 500.000 files with 170 GiB of data in total. So for each backup run, gpg is forked 500.000 times. I'm actually not even surprised that this takes almost an entire day for the initial run and over three hours for another run with most files unchanged. But is this really the performance obnam users should expect? For comparison: an incremental run of rsnapshot (same data, same machine, same drives) takes about four minutes. Granted, there is no encryption involved, but this shouldn't be that significant.
So, to ask plainly: Is everyone else's machine not able to run gpg (encrypting a small chunk of data) more than 50 times per second either, ultimately making obnam an almost unusably slow tool? Or is it just me?
(FWIW, my machine is a Core i5-2500 with 8G of RAM and SSD drives, running Gentoo. Backup is done onto an HDD, but I couldn't observe any difference to backing up to the SSD, since it's not I/O-bound.)
Best Answer
Here's a good read on how to speed up obnam (may run up to 10 times faster): http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html
Summary: add "--lru-size=1024 --upload-queue-size=512" to your commandline or config file. Note that it increases obnam's memory usage a bit.