Debian Package Management – How to Replicate Installed Package Selections

debiandpkgpackage-management

I am trying to set up a staging environment in a VM, in order to test updates before applying them to my main system.

In order to do so, I have done a basic installation of Debian Wheezy (same as on the main system) in the VM, then ran as root from within the VM:

# dpkg --clear-selections
# dpkg --add-architecture i386
# apt-get update
# ssh me@main-system 'dpkg --get-selections | grep -v deinstall' | \
  dpkg --set-selections

The i386 architecture is unfortunately needed in my case; the system is amd64 native.

The problem is with dpkg --set-selections run in the VM. I do have some packages that require special handling (those are actually the main reason why I want a staging environment in the first place) but when I run the last command above, I get about a gazillion lines of output like:

dpkg: warning: package not in database at line NNN: package-name

for packages that really should be available in the base system. Examples include xterm, yelp and zip.

Now for my question:

What is the specific process for transferring the package selection list from one Debian system to another (assuming same Debian release level, in Wheezy) and then subsequently applying those changes? The goal is that both have the same list of installed packages, ideally such that doing a diff between the outputs of dpkg --get-selections or dpkg --list on the two comes back showing no differences.

The grep -v deinstall part is borrowed from Prevent packages from being removed after doing dpkg --set-selections over on Ask Ubuntu.

I have changed the source in the VM to be the same as on the main system, also installing apt-transport-https:

deb https://ftp-stud.hs-esslingen.de/debian/ wheezy main non-free
deb-src https://ftp-stud.hs-esslingen.de/debian/ wheezy main non-free
deb https://ftp-stud.hs-esslingen.de/debian/ wheezy-updates main non-free
deb-src https://ftp-stud.hs-esslingen.de/debian/ wheezy-updates main non-free
deb [arch=amd64] http://archive.zfsonlinux.org/debian wheezy main

Looking at the –set-selections output, I'm seeing:

dpkg: warning: package not in database at line 1: a2ps
dpkg: warning: package not in database at line 1: abiword
dpkg: warning: package not in database at line 1: abiword-common
dpkg: warning: package not in database at line 1: abiword-plugin-grammar
dpkg: warning: package not in database at line 1: abiword-plugin-mathview
dpkg: warning: package not in database at line 1: accountsservice
dpkg: warning: package not in database at line 1: acl
dpkg: warning: package not in database at line 4: aglfn
dpkg: warning: package not in database at line 4: aisleriot
dpkg: warning: package not in database at line 4: alacarte
dpkg: warning: package not in database at line 4: alien
...

The line numbers looked odd, and the corresponding portion of the output of –get-selections is:

a2ps install
abiword install
abiword-common install
abiword-plugin-grammar install
abiword-plugin-mathview install
accountsservice install
acl install
acpi-support-base install
acpid install
adduser install
aglfn install
aisleriot install
alacarte install
alien install

Notice that in between acl and aglfn are acpi-support-base, acpid and adduser for which no errors are being reported. It seems that the packages for which errors are being reported are either un according to dpkg -l, or dpkg -l doesn't have any idea at all about them (dpkg-query: no packages found matching ...). I know there are some locally installed packages, but not many. i386 doesn't figure until gcc-4.7-base:i386 install much farther down the list (line 342 in the –get-selections output).

Best Answer

To clone a Debian installation, use the apt-clone utility. It's available (as a separate package, not part of the default installation) in Debian since wheezy and in Ubuntu since 12.04. On the existing machine, run

apt-clone clone foo

This creates a file foo.apt-clone.tar.gz. Copy it to the destination machine, and run

apt-get install apt-clone
apt-clone restore foo.apt-clone.tar.gz

If you're working with an old system where apt-clone isn't available, or if you just want to replicate the list of installed packages but not any configuration file, here are the manual steps.

On the source machine:

cat /etc/apt/sources.list /etc/apt/sources.list.d >sources.list
dpkg --get-selections >selections.list
apt-mark showauto >auto.list

On the target machine:

cp sources.list /etc/apt/
apt-get update
/usr/lib/dpkg/methods/apt/update /var/lib/dpkg/
dpkg --set-selections <selections.list
apt-get dselect-upgrade
xargs apt-mark auto <auto.list

I believe that you're affected by an incompatible change in dpkg that first made it into wheezy. See bug #703092 for background.

The short story is that dpkg --set-selections now only accepts package names that are present in the file /var/lib/dpkg/status or /var/lib/dpkg/available. If you only use APT to manage packages, like most people, then /var/lib/dpkg/available is not kept up-to-date.

After running apt-get update and before running dpkg --set-selections and apt-get -u dselect-upgrade, run the following command:

apt-cache dumpavail >/tmp/apt.avail
dpkg --merge-avail /tmp/apt.avail

From jessie onwards, you can simplify this to

apt-cache dumpavail | dpkg --merge-avail

Alternatively, run

/usr/lib/dpkg/methods/apt/update /var/lib/dpkg/

or even simpler

apt-get install dctrl-tools
sync-available

Another simple method that doesn't require installing an additional package but will download the package lists again is

dselect update

See the dpkg FAQ for more information. (This is mentioned in the dpkg man page, but more in a way that would remind you of the issue if you were already aware, not in a way that explains how to solve the problem!)

Note that cloning a package installation with dpkg --set-selections doesn't restore the automatic/manual mark in APT. See Restoring all data and dependencies from dpkg --set-selections '*' for more details. You can save the marks on the source system with

apt-mark showauto >auto.list

and restore them on the target system with

xargs apt-mark auto <auto.list

Related Solutions

Debian – How to keep the Debian system with the latest packages

Wanting more recent packages is a common issue on any OS. Debian's release cycle has averaged 2 years in recent years, so towards the end of this cycle, it is perhaps a more pressing issue. One way of mitigating this is to move to testing towards the end of the stable release cycle, when the next version is almost stable. It is not clear from the question whether it is talking about stable on more generally about testing and/or unstable as well. Regardless, having the most recent version can be an issue even if one if running unstable, as the most recent version may not be packaged yet. Debian developers/packagers are volunteers, so they may get bored, or busy with other things, with the result that the package languishes.

For simplicity and concreteness I assume in what follows that the plan is to backport a package to stable, but it applies more generally. So, here is what I do if I want a more recent version of software that is not present in stable, in an approximate order.

Look for the package in Debian Backports. Sometimes you can find a package that is recent enough to satisfy your purposes. However, it is often the case that these packages are outdated compared to the version in unstable or experimental or upstream.
Try to install the package directly from testing, unstable, or experimental. If stable has not diverged much from whatever version you are trying to install from, this may work. You'll know this approach is a bad one if the system starts trying to install or upgrade basic packages from the more recent version. Suppose you are trying to install from unstable, then
```
apt-get install packagename/unstable
```
is the first thing to try. With versions of apt in stable, this will often fail, as it requires other packages from unstable, and this incantation only ups the preference of packagename sufficiently high for it to be installed in unstable. If you don't understand what this means, go away and read man apt_preferences. Carry on adding dependencies from unstable, making sure that it is not trying to upgrade basic packages. For example, if it starts trying to upgrade libc6 or X or KDE or Gnome, abort immediately. It is usually fine if it is tries to upgrade other packages from the same source package, as these are usually tightly coupled together. To see what source package a binary package depends on, do
```
apt-cache showsrc packagename
```
Since lots of stuff depends on the GNU C library (libc6) this used to be a problem. More recently, the API seems to have stabilized, so it is now more often possible to get away with not having to upgrade it. If a package satisfies its runtime dependencies on stable, but still does not work correctly, file a bug. If the packager tells you it is not a bug, they are wrong. :-)
Backport the package yourself from testing, unstable or experimental.

As mentioned above, backports is one option, but often these packages are outdated compared to the version in unstable or experimental or upstream.

This can often require a recursive dependency build loop type thing. You first need to get the build dependencies with
```
apt-get build-dep packagename    
```
If this fails because one of the dependencies is not recent enough, you'll need to backport that dependency first. This can spriral out of control. I usually give up if I have to deal with more than 2 levels of recursion. Note however, that the real dependencies aren't necessarily as tight as stated ie. an older version may work. The packager often don't try to find the oldest version of a build (or, indeed, runtime) dependency that will work.
Check for the availability of packages from the corresponding upstream. Ideally these would match your distribution version, but you might also be able to rebuild them if necessary.
Create packages for version of the software more recent than the most recent packages in testing/unstable/experimental. This can be relatively challenging, but still sometimes surprisingly doable. The first thing to note is that if you are trying to package a more recent version of a package that is already in Debian, you are already starting with a big advantage, namely that you have the existing packaging to work with. Just do
```
apt-get source packagename
```
and apt-get will download the corresponding source package, including the debian subdirectory where the packaging lives. Note furthermore that these days, this packaging often lives inside some verson control repository (git seems popular with Debian) and stable apt (currently 0.8.10.3) helpfully tells you where this is when you invoke apt-get source. You should look at this, because the packagers may have more recent versions of the packaging than corresponds to any released package. Eg.
```
$ apt-get source mercurial
  Reading package lists... Done
  Building dependency tree       
  Reading state information... Done
  NOTICE: 'mercurial' packaging is maintained in the 'Svn' version control system at:
  svn://svn.debian.org/python-apps/packages/mercurial/trunk
```
Alternatively, you could simply use
```
apt-cache showsrc mercurial | grep Vcs
```
to list the repository.

If the package is badly out of date, you may have to make modifications to the
package, refresh the applied patches but it is still usually a good starting
point. Debian seems to be in the process of standardizing package management on
quilt per the dpkg-source 3.0 (quilt) format, so that helps with patch refreshing.

I'll conclude with a real-life example of how I backported the Debian package of pgf. The last packaged version of pgf was 2.00 in 2008, and since then 2.10 had been released. See the discussion in Please update to newest stable version of pgf (2.10), and my followup bug with a patch, pgf: patches against 2.0 Debian packaging. As it turns out, the Debian packaging of pgf was very simple, and I just had to change one line in the 2.10 packaging to make it work. I ended up quelling all the lintian complaints as well, but that was strictly optional.

Debian – How to install Debian package with file extension .deb

As pointed out in the comments, if that's an exact quote of what you're typing to try to install it you're making a typo:

sudo dkpg -i file.deb

won't work, but

sudo dpkg -i file.deb

will.

Spelling is not optional when it comes to the command line.

Best Answer

Related Solutions

Debian – How to keep the Debian system with the latest packages

Debian – How to install Debian package with file extension .deb

Related Question