How Does apt-get Work in Detail?

aptpackage-managementsoftware installation

I'm currently working with Ubuntu and trying to install a Debian system on a VM, which means I have to deal with packages. Since I'm new to it, I've read a lot about it on the web and especially debian wiki (ubuntu wiki is useful too). In particular, I have learn how to make a local repository and how to properly edit the sources.list file.

However, I still have some questions about how it works, details I couldn't find on the web.

First, when you use an online repository. I understood that apt-get will follow the link you wrote in the source.list file and search for a Packages.gz archive. What i don't understand is how this simple archive (which seem to be only a list of installable packages) allow the installation of the package? Does Packages.gz gives apt information about how to find the .deb file and then proceed the installation? Or is it something else?
Second, about the Sources.gz archive. I've read it's the source code of the packages listed in Packages.gz, BUT in most cases it's not needed. So, if i add the link to this Sources.gz in my source.list file, what does it really gives to me? What's the point of it?
Third, about local repository; this is related to the previous questions. I know how to make a local repository with .deb files, but let's say i only have this Packages.gz archive: it won't be enough, rigth? and if i have the Sources.gz archive, will it work?
Finally, i've seen on Debian repository that, in addition to the dist repository, there is a pool repository with a lot of .deb in it. I know a way to install those (download the .deb and its dependencies, install them with dpkg -i), but is there an easier way? Something more automatic, which could find the .deb online, find its dependencies, and install everything ? (just like apt-get do, but as far as i've understood it, apt-get only works with the Packages.gz and Sources.gz archives, not directly with .deb)

I'm sorry this is not really a question, rather a list of questions ^^ but everything I found online was mostly "apt-get search for the package and its dependency in the online repository and install it through dpkg", and I would like more details about it.

Best Answer

The Packages.gz contains a Filename field with a value that probably looks something like pool/main/n/name-of-package/name-of-package_version_amd64.deb. This tells the package manager to look at that URL.
For instance, if you have the following line in your sources.list(.d):
deb https://some-domain.com/some-url some-distribution main
You will have the following file: https://some-domain.com/some-url/dists/some-distribution/main/binary-amd64/Packages.gz
That file will reference pool/main/n/name-of-package/name-of-package_version_amd64.deb, so your package manager will look at https://some-domain.com/some-url/pool/main/n/name-of-package/name-of-package_version_amd64.deb.
Sources.gz works the same way, but is optional. It works with apt source, which will fetch you the source if you wish to compile it yourself instead of using the pre-built binaries.
This is mainly useful for system administrators who want to patch the exact version of the package used by a distribution, rather than contributors/curious people who would just go to the project's homepage and follow the build instructions there.
If a package is in the distribution's pool directory but not in Packages.gz, it is most probably in another Package.gz. Notice how you have several words in a typical sources.list line:
deb http://archive.ubuntu.com/ubuntu precise main universe multiverse
main, universe and multiverse each have their own Packages.gz file. You may need to enable some, as debian for instance disables non-free by default. However, some packages still won't be available (not even in pool). You may choose to add untrusted repositories (such as PPAs on Ubuntu), package the missing program yourself (so you and other people can use their package manager to manage that program) or simply build and install it yourself (you should use the /usr/local prefix to avoid conflicts with your package manager).
When a new (version of a) package appears, the Release file for the distribution is updated. This allows apt update to know it has to re-download Packages.gz.

Related Solutions

Ubuntu – Is it safe to manually perform ‘apt-get update’ ‘s operation

This sounds like it may work, but personally, I'd just use apt-offline.

From the manpage:

apt-offline brings offline package management functionality to Debian based system. It can be used to download packages and its dependencies to be installed later on (or required to update) a disconnected machine. Packages can be downloaded from a different connected machine.

Excerpt from Debian Administration:

Using apt-offline:

You generate a signature on your Debian box at home and carry the signature file on a removable medium
Now you take the USB Stick (with the apt-offline.txt signature file) to the office machine which could be running any linux version or even Windows.
There, you could run apt-offline giving it the signature file.
apt-offline would generate you an archive file or a folder with all the data. That data can be copied on a removable media. The removable media can be attached back to the disconnected Debian box at home and installed. (e.g. "apt-offline install /tmp/apt-offline.zip")

APT-GET – How APT-GET Really Works

You need to take a look at https://wiki.debian.org/Packaging — the packaging tutorial there will help you a lot, as well as parts of the new maintainer's guide.

As to your questions, in order:

The repository contains "list" files. E.g.., http://http.us.debian.org/debian/dists/stretch/main/binary-amd64/Packages.xz. apt-get update downloads these list files, and stores them in /var/lib/apt/lists. The list files list all the packages including a bunch of metadata and a relative URL to find the .deb at. (They're human-readable plain-text files, so you can just look at it).
OS doesn't matter. You could host it on Windows, if you wanted. (Well, you'd maybe have trouble with file names Windows doesn't like.) (See also #4 and #5).
Yes, it's inside the deb file. A deb file is actually an archive (using ar). Inside are some tar files; one of them is (essentially) extracted to /.
It's just HTTP (or HTTPS, or FTP, or... apt-get supports a lot of protocols). Nothing special, though. Note that there are Release files, signed with gpg, which guarantee integrity even w/o HTTPS. Debian mirrors mostly use HTTP, not HTTPs. (A few support HTTPS as well for confidentiality).
It's just a structured filesystem.

A quick, high-level overview of how apt-get interacts with a package source:

You configure which sources to look at in your sources.list file. Consider a line like:
```
deb http://http.us.debian.org/debian/ stretch main
```
deb says this is a source for gettings .deb (binary) files; then there is the URL-prefix, suite/release ("stretch"), and component ("main").
apt-get has a list of architectures, it gets that from dpkg. Let's say dpkg --print-architecture is amd64. apt-get can now build the URLs its actually going to download from, by combining the URL-prefix, the word "dists", the suite, the component, and the architecture. Then it tacks on a few fixed filenames, like "Packages.xz". That gives the URL above (in #1). There are a few more files with defined names/paths, like the Release file http://http.us.debian.org/debian/dists/stretch/Release and its signature (same, with .gpg appended). These are all (possibly-compressed) plain-text files. The release file contains checksums for other files apt-get is going to download, like Packages.xz.
The Packages.xz file lists all the packages in that suite/codename/architecture. It also gives the path where that file is located; for example pool/main/0/0ad/0ad_0.0.21-2_amd64.deb.
When you ask apt-get to download a package, it uses that location + the base URL to download the package, so that package is at http://http.us.debian.org/debian/pool/main/0/0ad/0ad_0.0.21-2_amd64.deb
The other interesting directory is source instead of binary-amd64. That's used for your deb-src entries; it contains info about source packages (and is otherwise fairly similar).
There are some other things (all of them optional, I believe) that can be part of the repository (i.e., available via HTTP): diffs between different versions of the Packages.xz file; translations of package descriptions, a complete list of every installable file and which package it belongs to (Contents-amd64.gz, used by e.g., apt-file, not by apt-get) etc. These likely aren't relevant to you, but you can see them all by browsing around http://http.us.debian.org/debian/dists/stretch/; most of them are plain-text files.

All these files are plain text. They can, in theory, be created by hand. In practice, everyone uses one of these repository generation tools. Here—and I caution this was a choice made a long time ago, so may be outdated—we use mini-dinstall. The output of those tools are ordinary files or, at worst, symlinks. You can rsync them over to whatever web server you want.

Best Answer

Related Solutions

Ubuntu – Is it safe to manually perform ‘apt-get update’ ‘s operation

APT-GET – How APT-GET Really Works

Related Question