Ubuntu – Improve full backup speed with Déjà Dup / Duplicity

backupdeja-dupduplicitygnome

Is there an ease fix two to improve the backup speed using the default Déjà Dup Backup solution of the Ubuntu Gnome Desktop?

Actually I'm quite happy with my weekly duplicity backups, but I set it up that I have every three month a full backup to my local NAS. This takes around two full days for ~1TB of data.

Deja Dup Backup

I believe there is at least two bottlenecks of my duplicity solution:

  1. the backup runs only one core and
  2. it uploads just small chunks of 50 MB to the NAS.

Especially the second issue means, that I can't use the full capacity of my Gigabit-LAN. Duplicity uploads the files with about 25 MiB/s. I upload manually a bigger file I get easily 80 MiB/s (NAS is connected with a 2 x 1 GbE bond).

Here is a typical screenshot of the System Monitor:
screenshot system monitor gnome

Does anybody have an idea, how to improve the full backup speed? Eg. can I change the chunk size etc.

Best Answer

Found it, Deja Dup overwrites the default duplicity chunk size. The source code explains the advantages and disadvantages:

  // Returns volume size in megs
  int get_volsize()
  {
    // Advantages of a smaller value:
    // * takes less temp space
    // * retries of a volume take less time
    // * quicker restore of a particular file (less excess baggage to download)
    // * we get feedback more frequently (duplicity only gives us a progress
    //   report at the end of a volume) -- fixed by reporting when we're uploading
    // Downsides:
    // * less throughput:
    //   * some protocols have large per-file overhead (like sftp)
    //   * the network doesn't have time to ramp up to max tcp transfer speed per
    //     file
    // * lots of files looks ugly to users
    //
    // duplicity's default is 25 (used to be 5).
    //
    // For local filesystems, we'll choose large volsize.
    // For remote FSs, we'll go smaller.
    if (DejaDup.in_testing_mode())
      return 1;
    else if (backend.is_native())
      return 50;
    else
      return 25;
  }

I managed to compile a custom version of deja-dup. Therefore, I had to do the following steps:

  1. Clone source code

    git clone https://gitlab.gnome.org/World/deja-dup.git
cd ./deja-dup

  2. Switch to older branch with gtk3 support for Ubuntu 20.04

    git checkout --track origin/40

  3. Patch the return of the get_volsize() funktion in DuplicityJob.vala and increase the version number. Eg.

    git apply patch40-7-1.diff

    patch40-7-1.diff:

    diff --git a/libdeja/tools/duplicity/DuplicityJob.vala b/libdeja/tools/duplicity/DuplicityJob.vala
index a229e8b0..bb6f77fe 100644
--- a/libdeja/tools/duplicity/DuplicityJob.vala
+++ b/libdeja/tools/duplicity/DuplicityJob.vala
@@ -1399,9 +1399,9 @@ internal class DuplicityJob : DejaDup.ToolJob
     if (DejaDup.in_testing_mode())
       return 1;
     else if (backend.is_native())
-      return 50;
+      return 500;
     else
-      return 25;
+      return 250;
   }

   void disconnect_inst()
diff --git a/meson.build b/meson.build
index 266d7a36..fed8434d 100644
--- a/meson.build
+++ b/meson.build
@@ -4,7 +4,7 @@
 # SPDX-FileCopyrightText: Michael Terry

 project('deja-dup', ['vala', 'c'],
-    version: '40.7',
+    version: '40.7.1',
     license: 'GPL-3.0-or-later',
     meson_version: '>= 0.47'

  4. Make and install (including installing missing packages):

    meson . _build
ninja -C _build
ninja -C _build install

Voilà

custom compiled deja dup

Related Question