Configuration management via versioned packages

ansibleconfigurationdebpackage-management

For the hosts we manage, my boss has a vision of configuration management via versioned packages (in our particular case, fpm-built debian packages). This is to make:

host deployment easier
easier to track local changes
make bug tracking easier as in most cases, we can tag a version of software to a bug, whether it is a functional or configuration bug.

The other alternative is configuration management via Ansible, with their recommended layout as in their manual, which we are using currently. We install versioned packages of software (pretty much all web services), which would be responsible for it's own package-specific configuration, and leave Ansible to system-wide configuration. Updating hosts software is simply a matter of bumping version numbers and rerunning the scripts. We rely heavily on the "var_prompts" functionality of Ansible to ensure we keep sensitive information separate from the configuration.

In the "configuration management via versioned packages" case:

For deployment, we just bring up a minimal server (proxies etc) and use a package manager (apt in our case) to install a single umbrella package, that would install all the dependent packages. Making a change to any of the dependent packages would simply bump up the top most package version, making it easier to update existing deployments.

For configuration, each software package would have an accompanying "config" package, to allow changes to configuration only. Each package would have independent configuration files, to prevent conflicts between packages when we verify local changes on the host. Instead of running commands that generate config, we would ensure that we would have the generated files as part of the package, but still allow running of commands via the pre/post install scripts. AFAIK, there aren't a great number of cases where we have packages that will rely on the same configration file, but if that is the case (which would be the case for system configuration), we would have another package that would have a dependency on both. I can see this particular case getting out of control, but this could be controlled by always ensuring independent config files. In the end, each host would have their own unique config package. Obviously, all these files would be under source control, and we may spin up our own simple templating system, so that we can also have the flexibility of Ansible roles.

I am curious why I cannot find any information/analysis on this type of workflow for configuring and deploying hosts. Am I missing something fundamentally flawed with such an approach, in comparison to existing configuration management tools?

Best Answer

Package managers are not really effective for configuration managment for a few reasons:

Most packages contain a configuration when installed, so you'd have to make custom packages that have them removed.
Package managers work under the assumption that the package installs a basic configuration that can be changed within the system. So if a package is reinstalled/upgraded, it usually prompts the user on what to do with the modified file (which is not an ideal behavior for a configuration manager).
Package manager do not test the system against the package contents usually, and although some have that option, it's not efficient as it's not a feature that is often used.
A package life-cycle is more complex and longer than most of the automated configuration tools, so it's more work to maintain such a setup.
Package managers do not care about order of installation so if you have a case of package dependency of a->b->c and you have package c installed already and you install package a, package c will not be reinstalled and packages a and/or b may overwrite c's configuration.

If you insist on using package managers to maintain configurations, I'd suggest you'll check the option of using a standalone configuration management system like puppet or chef in their standalone mode with a package for their configuration, this way you'll have all of your configuration in a single versioned package but in a way that does not require the package manager to manage the configurations on disk, which can lead to problems as I listed and probably some more.

To clarify, it should be possible to do what you planned, just the package manager is not the ideal tool for this.

Related Solutions

Debian – How to keep the Debian system with the latest packages

Wanting more recent packages is a common issue on any OS. Debian's release cycle has averaged 2 years in recent years, so towards the end of this cycle, it is perhaps a more pressing issue. One way of mitigating this is to move to testing towards the end of the stable release cycle, when the next version is almost stable. It is not clear from the question whether it is talking about stable on more generally about testing and/or unstable as well. Regardless, having the most recent version can be an issue even if one if running unstable, as the most recent version may not be packaged yet. Debian developers/packagers are volunteers, so they may get bored, or busy with other things, with the result that the package languishes.

For simplicity and concreteness I assume in what follows that the plan is to backport a package to stable, but it applies more generally. So, here is what I do if I want a more recent version of software that is not present in stable, in an approximate order.

Look for the package in Debian Backports. Sometimes you can find a package that is recent enough to satisfy your purposes. However, it is often the case that these packages are outdated compared to the version in unstable or experimental or upstream.
Try to install the package directly from testing, unstable, or experimental. If stable has not diverged much from whatever version you are trying to install from, this may work. You'll know this approach is a bad one if the system starts trying to install or upgrade basic packages from the more recent version. Suppose you are trying to install from unstable, then
```
apt-get install packagename/unstable
```
is the first thing to try. With versions of apt in stable, this will often fail, as it requires other packages from unstable, and this incantation only ups the preference of packagename sufficiently high for it to be installed in unstable. If you don't understand what this means, go away and read man apt_preferences. Carry on adding dependencies from unstable, making sure that it is not trying to upgrade basic packages. For example, if it starts trying to upgrade libc6 or X or KDE or Gnome, abort immediately. It is usually fine if it is tries to upgrade other packages from the same source package, as these are usually tightly coupled together. To see what source package a binary package depends on, do
```
apt-cache showsrc packagename
```
Since lots of stuff depends on the GNU C library (libc6) this used to be a problem. More recently, the API seems to have stabilized, so it is now more often possible to get away with not having to upgrade it. If a package satisfies its runtime dependencies on stable, but still does not work correctly, file a bug. If the packager tells you it is not a bug, they are wrong. :-)
Backport the package yourself from testing, unstable or experimental.

As mentioned above, backports is one option, but often these packages are outdated compared to the version in unstable or experimental or upstream.

This can often require a recursive dependency build loop type thing. You first need to get the build dependencies with
```
apt-get build-dep packagename    
```
If this fails because one of the dependencies is not recent enough, you'll need to backport that dependency first. This can spriral out of control. I usually give up if I have to deal with more than 2 levels of recursion. Note however, that the real dependencies aren't necessarily as tight as stated ie. an older version may work. The packager often don't try to find the oldest version of a build (or, indeed, runtime) dependency that will work.
Check for the availability of packages from the corresponding upstream. Ideally these would match your distribution version, but you might also be able to rebuild them if necessary.
Create packages for version of the software more recent than the most recent packages in testing/unstable/experimental. This can be relatively challenging, but still sometimes surprisingly doable. The first thing to note is that if you are trying to package a more recent version of a package that is already in Debian, you are already starting with a big advantage, namely that you have the existing packaging to work with. Just do
```
apt-get source packagename
```
and apt-get will download the corresponding source package, including the debian subdirectory where the packaging lives. Note furthermore that these days, this packaging often lives inside some verson control repository (git seems popular with Debian) and stable apt (currently 0.8.10.3) helpfully tells you where this is when you invoke apt-get source. You should look at this, because the packagers may have more recent versions of the packaging than corresponds to any released package. Eg.
```
$ apt-get source mercurial
  Reading package lists... Done
  Building dependency tree       
  Reading state information... Done
  NOTICE: 'mercurial' packaging is maintained in the 'Svn' version control system at:
  svn://svn.debian.org/python-apps/packages/mercurial/trunk
```
Alternatively, you could simply use
```
apt-cache showsrc mercurial | grep Vcs
```
to list the repository.

If the package is badly out of date, you may have to make modifications to the
package, refresh the applied patches but it is still usually a good starting
point. Debian seems to be in the process of standardizing package management on
quilt per the dpkg-source 3.0 (quilt) format, so that helps with patch refreshing.

I'll conclude with a real-life example of how I backported the Debian package of pgf. The last packaged version of pgf was 2.00 in 2008, and since then 2.10 had been released. See the discussion in Please update to newest stable version of pgf (2.10), and my followup bug with a patch, pgf: patches against 2.0 Debian packaging. As it turns out, the Debian packaging of pgf was very simple, and I just had to change one line in the 2.10 packaging to make it work. I ended up quelling all the lintian complaints as well, but that was strictly optional.

Debian – Creating a Debian package repository for distributing multiple versions of a custom deb package

After spending some time digging around the a number of documentation sources, I came up with the following solution, which I am sharing:

On the machine used to host the repository, set up a personal repository according to the description of Ubuntu documentation, but do NOT add the sources.list entry on that machine.
Expose the directory with the DEB files and the Packages.gz file via a web server, e.g. with Nginx so that it is available all the machines that will consume the DEB file.
On the machines, which will fetch the DEB package, add an entry to the /etc/apt/sources.list file, which points to your server (replace foobar with your own URL): deb [trusted=yes] http://foobar/ /
Run sudo apt update
Run sudo apt install foobar to install your own package (replace foobar with your own package name)

NOTE: this setup hosts the DEB packages without any protection provided by signed packages/repositories. In case the repository is made available for a wider audience than your team's servers residing on a protected subnet behind a company firewall, you probably want to implement signing of the repository and packages.

Best Answer

Related Solutions

Debian – How to keep the Debian system with the latest packages

Debian – Creating a Debian package repository for distributing multiple versions of a custom deb package

Related Question