Python – Pip vs Package Manager for Handling Python Packages

aptpackage-managementpacmanpippython

Python packages are frequently hosted in many distribution's repositories. After reading this tutorial, specifically the section titled "Do you really want to do this" I have avoided using pip and preferred to use the system repository, only resorting to pip when I need to install a package not in the repository.

However, because this is an inconsistent installation method, would it be better to only use pip? What are the benefits/detractors to using pip over the system's own repository for packages that are available in both places?

The link I included states

The advantage of always using standard Debian / NeuroDebian packages, is that the packages are carefully tested to be compatible with each other. The Debian packages record dependencies with other libraries so you will always get the libraries you need as part of the install.

I use arch. Is this the case with other package-management systems besides apt?

Best Answer

The biggest disadvantage I see with using pip to install Python modules on your system, either as system modules or as user modules, is that your distribution’s package management system won’t know about them. This means that they won’t be used for any other package which needs them, and which you may want to install in the future (or which might start using one of those modules following an upgrade); you’ll then end up with both pip- and distribution-managed versions of the modules, which can cause issues (I ran into yet another instance of this recently). So your question ends up being an all-or-nothing proposition: if you only use pip for Python modules, you can no longer use your distribution’s package manager for anything which wants to use a Python module...

The general advice given in the page you linked to is very good: try to use your distribution’s packages as far as possible, only use pip for modules which aren’t packaged, and when you do, do so in your user setup and not system-wide. Use virtual environments as far as possible, in particular for module development. Especially on Arch, you shouldn’t run into issues caused by older modules; even on distributions where that can be a problem, virtual environments deal with it quite readily.

It’s always worth considering that a distribution’s library and module packages are packaged primarily for the use of other packages in the distribution; having them around is a nice side-effect for development using those libraries and modules, but that’s not the primary use-case.

Related Solutions

Debian – How to Install Packages from ‘Unstable’ on a ‘Stable’ System

You do need to have unstable listed in your sources.list. Otherwise apt just won't find the package.

To avoid unstable packages being pulled in, you have two ways.

The easy way is to add a Default-Release clause to /etc/apt/apt.conf (or to a file under /etc/apt/apt.conf.d/, e.g. /etc/apt/apt.conf.d/my-default-release).
```
APT::Default-Release "stable";
```

The hard way is to use APT preferences. In /etc/apt/preferences:

Package: *
Pin: release o=Debian,a=unstable
Pin-Priority: 10

Note that for most of the lifetime of a Debian release, it's not practical to install most packages from unstable on a stable system, because they'll pull in a lot of libraries from unstable, and you'll end up with an unstable system. If you want to run unstable, it'll save you trouble to just target unstable (or testing). It's best to stick to a single release, or testing with the occasional unstable if you're feeling daring. And of course, on a production system, stick to stable.

If you're running stable but you need a newer version of one application, first look if there is a backport for them. Otherwise, if you want to install a package from unstable but not have to pull in its dependencies, try getting the source from unstable and recompiling.

apt-get source foo=1.42
apt-get build-dep foo  # pulls the dependencies of foo in stable but that's often good enough
dpkg-source -x foo_1.42.dsc
cd foo-1.4210126#10126
dpkg-buildpackage -rfakeroot -us -uc -b -nc
dpkg -i ../foo_1.42_$(arch).deb

Debian – Creating a Debian package repository for distributing multiple versions of a custom deb package

After spending some time digging around the a number of documentation sources, I came up with the following solution, which I am sharing:

On the machine used to host the repository, set up a personal repository according to the description of Ubuntu documentation, but do NOT add the sources.list entry on that machine.
Expose the directory with the DEB files and the Packages.gz file via a web server, e.g. with Nginx so that it is available all the machines that will consume the DEB file.
On the machines, which will fetch the DEB package, add an entry to the /etc/apt/sources.list file, which points to your server (replace foobar with your own URL): deb [trusted=yes] http://foobar/ /
Run sudo apt update
Run sudo apt install foobar to install your own package (replace foobar with your own package name)

NOTE: this setup hosts the DEB packages without any protection provided by signed packages/repositories. In case the repository is made available for a wider audience than your team's servers residing on a protected subnet behind a company firewall, you probably want to implement signing of the repository and packages.

Best Answer

Related Solutions

Debian – How to Install Packages from ‘Unstable’ on a ‘Stable’ System

Debian – Creating a Debian package repository for distributing multiple versions of a custom deb package

Related Question