Apt forgot about more than 1,000 packages, and I’m unable to reinstall

aptaptitudedpkgpackage-managementsoftware installation

I'm running 20.04.2 LTS (Focal Fossa)

I hadn't run aptitude for a couple days. It always worked fine, but when I tried it now, it started giving me the "e Examine" dialog each time I run it. When I type "e", it goes into a 10-20 minute run with the following showing at the top of the page:

open: aaa; closed: bbb; defer: ccc; conflict: ddd

Where 'aaa', 'bbb', 'ccc', 'ddd' are increasing numeric values … and 'aaa' and 'bbb' are increasing rapidly, up to more than 50,000.

After a while, aptitude simply exits.

At this point, if I run dpkg --configure -a, it quickly returns to the command prompt with no error. But then, when I run apt-get install -f, I get a list of more than 1,000 package names followed by this …

Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Most of these packages (probably all of them, actually) are items that I use from day to day, and they should not be removed.

Oh, also, if I capture the huge list of packages that are shown when I do apt-get install -f, and if I generate a thousand or so apt install PKG commands for each of these package names and then run this, I just get multiple lists of those same thousand-or-so packages for each of these commands, followed by that same "use sudo apt autoremove" message after each apt install PKG run.

Somehow, my apt caches seem to have gotten hosed. I don't know what could have caused this.

However, I don't really want to spend time trying to figure out why this happened. I just want to fix it so that all those packages are once again properly installed and manageable via apt.

Can anyone point me to docs that could help me, or perhaps to suggest a solution?

Thank you very much in advance.

PS: my /etc/apt/sources.list hasn't changed for weeks, nor have any files in /etc/apt/source.list.d changed in a long time. Actually, nothing under /etc/apt seems to have changed since a couple months ago.

PPS*: Here are the last dozen or so lines of /var/log/apt/history.log …

Start-Date: 2021-07-24  20:01:24
Requested-By: ljz (1000)
Install: linux-headers-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic), linux-modules-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic), linux-headers-5.4.0-80:amd64 (5.4.0-80.90, automatic), linux-image-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic), linux-modules-extra-5.4.0-80-generic:amd64 (5.4.0-80.90, automatic)
Upgrade: networkd-dispatcher:amd64 (2.0.1-1, 2.1-2~ubuntu20.04.1), linux-headers-generic:amd64 (5.4.0.77.80, 5.4.0.80.84), linux-libc-dev:amd64 (5.4.0-77.86, 5.4.0-80.90), libcurl4:amd64 (7.68.0-1ubuntu2.5, 7.68.0-1ubuntu2.6), ruby2.7:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), libsystemd0:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), linux-image-generic:amd64 (5.4.0.77.80, 5.4.0.80.84), libsystemd-dev:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), python3-pip:amd64 (20.0.2-5ubuntu1.5, 20.0.2-5ubuntu1.6), udev:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), ruby2.7-dev:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), ruby2.7-doc:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), libudev1:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), systemd-timesyncd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libxnvctrl0:amd64 (460.73.01-0ubuntu0.20.04.1, 470.57.01-0ubuntu0.20.04.1), libglib2.0-dev-bin:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libruby2.7:amd64 (2.7.0-5ubuntu1.4, 2.7.0-5ubuntu1.5), systemd-sysv:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libglib2.0-bin:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libpam-systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libglib2.0-dev:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libglib2.0-doc:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libnss-systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.11), libglib2.0-data:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), curl:amd64 (7.68.0-1ubuntu2.5, 7.68.0-1ubuntu2.6), linux-generic:amd64 (5.4.0.77.80, 5.4.0.80.84), python-pip-whl:amd64 (20.0.2-5ubuntu1.5, 20.0.2-5ubuntu1.6), libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.3, 2.64.6-1~ubuntu20.04.4), libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.5, 7.68.0-1ubuntu2.6)
End-Date: 2021-07-24  20:02:46

Start-Date: 2021-07-25  19:31:25
Requested-By: ljz (1000)
Remove: linux-modules-extra-5.4.0-74-generic:amd64 (5.4.0-74.83), linux-modules-extra-5.4.0-73-generic:amd64 (5.4.0-73.82)
End-Date: 2021-07-25  19:31:28

Start-Date: 2021-07-27  18:56:53
Requested-By: ljz (1000)
Upgrade: libaspell15:amd64 (0.60.8-1build1, 0.60.8-1ubuntu0.1), aspell:amd64 (0.60.8-1build1, 0.60.8-1ubuntu0.1), libmysqlclient21:amd64 (8.0.25-0ubuntu0.20.04.1, 8.0.26-0ubuntu0.20.04.2), teamviewer:amd64 (15.19.3, 15.20.3)
End-Date: 2021-07-27  18:57:05

aptitude worked fine around 2021-07-24, so whatever broke my setup must have occurred after that.

Output of sudo aptitude search ?broken

iB  anydesk                                                 - The fastest remote desktop software on the market.                
iBA apt                                                     - commandline package manager                                       
iBA aptitude                                                - terminal-based package manager                                    
iBA base-files                                              - Debian base system miscellaneous files                            
iBA base-passwd                                             - Debian base system master password and group files                
iBA bash                                                    - GNU Bourne Again SHell                                            
iBA bsdutils                                                - basic utilities from 4.4BSD-Lite                                  
iB  chromium-common                                         - web browser - common resources used by the chromium packages      
iB  chromium-sandbox                                        - web browser - setuid security sandbox for chromium                
iB  containerd.io                                           - An open and reliable container runtime                            
iBA coreutils                                               - GNU core utilities                                                
iBA dash                                                    - POSIX-compliant shell                                             
iBA debianutils                                             - Miscellaneous utilities specific to Debian                        
iBA diffutils                                               - File comparison utilities                                         
iBA dpkg                                                    - Debian package management system                                  
iBA e2fsprogs                                               - ext2/ext3/ext4 file system utilities                              
iBA fdisk                                                   - collection of partitioning utilities                              
iBA findutils                                               - utilities for finding files--find, xargs                          
iB  google-chrome-stable                                    - The web browser from Google                                       
iBA grep                                                    - GNU grep, egrep and fgrep                                         
iBA gzip                                                    - GNU compression utilities                                         
iBA hostname                                                - utility to set/show the host name or domain name                  
iBA libc-bin                                                - GNU C Library: Binaries                                           
iBA libcurl3-gnutls                                         - easy-to-use client-side URL transfer library (GnuTLS flavour)     
iBA libglib2.0-0                                            - GLib library of C routines                                        
iBA libnss-systemd                                          - nss module providing dynamic user and group name resolution       
iBA libpam-systemd                                          - system and service manager - PAM module                           
iBA libsystemd0                                             - systemd utility library                                           
iBA libudev1                                                - libudev shared library                                            
iBA linux-base                                              - Linux image base package                                          
iBA linux-headers-5.4.0-80-generic                          - Linux kernel headers for version 5.4.0 on 64 bit x86 SMP          
iBA linux-image-5.4.0-80-generic                            - Signed kernel image generic                                       
iBA linux-image-generic                                     - Generic Linux kernel image                                        
iBA linux-modules-extra-5.4.0-80-generic                    - Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP    
iBA login                                                   - system login tools                                                
iBA mount                                                   - tools for mounting and manipulating filesystems                   
iBA ncurses-bin                                             - terminal-related programs and man pages                           
iB  net.downloadhelper.coapp                                - Video DownloadHelper companion app                                
iBA networkd-dispatcher                                     - Dispatcher service for systemd-networkd connection status changes 
iB  ocenaudio                                               - Ocenaudio is a fast, cross-platform audio editor.                 
iB  palemoon                                                - Firefox-based, efficient and easy to use web browser              
iBA perl-base                                               - minimal Perl system                                               
iB  rpi-imager                                              - Raspberry Pi imaging utility                                      
iBA sed                                                     - GNU stream editor for filtering/transforming text                 
iB  signal-desktop                                          - Private messaging from your desktop                               
iB  signal-desktop-beta                                     - Private messaging from your desktop                               
iBA systemd                                                 - system and service manager                                        
iBA systemd-timesyncd                                       - minimalistic service to synchronize local time with NTP servers   
iBA sysvinit-utils                                          - System-V-like utilities                                           
iBA tar                                                     - GNU version of the tar archiving utility                          
iBA util-linux                                              - miscellaneous system utilities                                    
iB  vdpau-va-driver                                         - VDPAU-based backend for VA API                                    
iB  zoom                                                    - Zoom, #1 Video Conferencing and Web Conferencing Service

And here's more:

% sudo apt update
Hit:1 https://linux.teamviewer.com/deb stable InRelease
Hit:2 https://updates.signal.org/desktop/apt xenial InRelease                                     
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]                         
Hit:4 http://us.archive.ubuntu.com/ubuntu focal InRelease                                                    
Hit:5 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease                                            
Get:6 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Hit:7 http://archive.canonical.com/ubuntu focal InRelease                                
Fetched 214 kB in 1s (178 kB/s)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.

% sudo apt list --upgradeable
Listing... Done
signal-desktop-beta/xenial 5.11.0-beta.3 amd64 [upgradable from: 5.8.0-beta.1]
signal-desktop/xenial 5.10.0 amd64 [upgradable from: 5.9.0]
ubuntu-advantage-tools/focal-updates 27.2.2~20.04.1 amd64 [upgradable from: 27.1~20.0

% sudo apt-get install -f
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  fonts-ubuntu fonts-uralic fotoxx fotoxx-common friendly-recovery ftp gdal-data gegl genisoimage geoclue-2.0 geoip-database
  [ ... more than 100 lines containing more package names ... ]
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Still more …

grep -r "^deb" /etc/apt/ --include=*.list
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
/etc/apt/sources.list:deb-src http://us.archive.ubuntu.com/ubuntu/ focal restricted main universe multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
/etc/apt/sources.list:deb-src http://us.archive.ubuntu.com/ubuntu/ focal-updates restricted main universe multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal universe
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
/etc/apt/sources.list:deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
/etc/apt/sources.list:deb-src http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
/etc/apt/sources.list:deb http://archive.canonical.com/ubuntu focal partner
/etc/apt/sources.list:deb-src http://archive.canonical.com/ubuntu focal partner
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu focal-security main restricted
/etc/apt/sources.list:deb-src http://security.ubuntu.com/ubuntu focal-security restricted main universe multiverse
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu focal-security universe
/etc/apt/sources.list:deb http://security.ubuntu.com/ubuntu focal-security multiverse
/etc/apt/sources.list.d/teamviewer.list:deb https://linux.teamviewer.com/deb stable main
/etc/apt/sources.list.d/signal-xenial.list:deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main

Possible solution …

What if I do the following? …

  1. run sudo apt-get install -f >apt-installs
  2. edit that file, removing the lines that precede and follow the package names, and putting "sudo apt install " at the beginning of each line
  3. run "sudo apt autoremove"
  4. run "bash apt-installs" to reinstall each and every package.

Here's what the "apt-installs" file would look like after step 2 …

sudo apt install accountsservice acpi-support acpid adb alsa-base android-libadb android-libbase android-libboringssl android-libcrypto-utils
sudo apt install android-libcutils android-liblog android-sdk-platform-tools-common apt-rdepends aptdaemon aptdaemon-data apturl apturl-common
sudo apt install aria2 audacious audacious-plugins audacious-plugins-data bind9-dnsutils blueman bluez bluez-cups brltty brltty-x11 brotli
sudo apt install busybox-static byobu catdoc catfish cheese-common cinnamon-desktop-data cinnamon-l10n clang-format-11 clang-tidy-11
sudo apt install clang-tools-11 command-not-found cppcheck cvs dc dctrl-tools debhelper debtree devilspie devilspie2 dh-autoreconf
sudo apt install dh-elpa-helper dh-strip-nondeterminism diffstat distro-info dmz-cursor-theme doc-base docbook-xml dwz efibootmgr elfutils
[ ... etc. ... for more than 100 lines ]

What do you folks think? Could this fix my problem?

More follow-up …

After more investigation about apt, I now see that the sudo apt autoremove will indeed uninstall all of those packages. That's over 1,000 packages, and the my system will indeed become unusuable after that. And it's unclear to me whether running all of those sudo apt install ... commands will even work, given that so much other software would have been uninstalled.

So, I'm still stuck. 🙁

I don't want to uninstall any software. I just want to get the apt caches back into a usable state, without changing any existing software installations.

Best Answer

I figured out how to fix this problem. I needed to remove and reinstall all of those 1,000-plus packages, and I did so one-by-one. And then I needed to reinstall aptitude. To acomplish this fix, I performed the following steps ...

Step 1:

% sudo apt-get install -f >apt-installs

After running this, the apt-installs file contained this data ...

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  activity-log-manager apg audacious audacious-plugins cppcheck cracklib-runtime dh-strip-nondeterminism docbook-xml
  [ ... more than 180 lines containing lists of package names ... ]
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Step 2:

I edited this apt-installs file to remove the top 4 lines and the bottom 2 lines, and I reformatted it so that there is one package name per line.

Step 3:

I prepended the string fixit to the beginning of each line.

Step 4:

I created a shell function called fixit to remove and then install the specified package, and I put that function at the top of the apt-installs file. Then, here's what the first dozen or so lines of apt-installs look like ...

fixit() {
  [[ $# -lt 1 ]] && return 0
  package="${1}"
  shift
  echo "### fixing ${package}"
  apt-get -y remove "${package}"
  apt-get -y install "${package}"
  echo ''
}
              
fixit activity-log-manager
fixit apg
fixit audacious
fixit audacious-plugins
fixit cppcheck
fixit cracklib-runtime
fixit dh-strip-nondeterminism
fixit docbook-xml
fixit gkbd-capplet
fixit gnome-bluetooth
fixit gnome-control-center-data
[ ... etc. ... more than 1,500 lines ]

Step 5:

I ran this:

% sudo bash apt-installs

This removed and reinstalled each of the packages.

The advantage of these piecemeal "remove/install" runs for each package is that only one, single package is uninstalled and then reinstalled at any given moment, instead of blowing away more than 1,500 pieces of software via an apt autoremove and thereby rendering my system un-runnable.

Step 6:

I then ran these two commands ...

% sudo apt-get purge aptitude
% sudo apt-get install aptitude

Before I did this, aptitude was still hanging and crashing, even after the reinstalls of all the packages. This purge/install fixed the aptitude crashes.

And now, apt, apt-get, and aptitude are all working fine again, and my system is still usable.

All's well that ends well.

Related Question