APT – Resolve Heavy Load by APT Unattended-Upgrade

aptunattended-upgrades

16.04 Xenial

APT automatic update check: enabled

APT automatic update download & install: disabled

No cron jobs listed below,

$ for user in $(cut -f1 -d: /etc/passwd); do sudo crontab -u $user -l; done

The system suddenly topped by heavy load pertaining to both CPU and Network IO from the aroused unattended-upgrade for several minutes.

Why could it happen when automatic update download & install were totally disabled?

Best Answer

The functionality on unattended-upgrades is automatically enabled after the installation of 16.04 Xenial Server. I mistakenly didn't really disable it so observed the unexpected behaviors above. Method Two here I had followed is unlikely effective.

To properly disable the automatic updates, we need change the configuation on /etc/apt/apt.conf.d/20auto-upgrades from:

APT::Periodic::Unattended-Upgrade "1";

to:

APT::Periodic::Unattended-Upgrade "0";

cron-apt

Cron-apt only downloads new packages by default. This can be useful if you have a strongly customized system. (for more details, see source [1])

unattended-upgrades

apt includes support for running an unattended upgrade script; Ubuntu includes this with the ''unattended-upgrade'' package which handles both security and non-security updates. The frequency of updates can be changed by setting the variable APT::Periodic::Unattended-Upgrade "x"; where x is the number of days.

This is the recommended way to do unattended upgrades in Ubuntu:

sudo apt-get install unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

(-plow is the short form of --priority=low. That is, show all questions of 'low' priority or higher.)

Automatic upgrades can break your system, so be aware, that you should only install this on systems where a failiure is not fatal to any means.

[1] Source: https://help.ubuntu.com/community/AutoWeeklyUpdateHowTo

Ubuntu – apt/unattended-upgrades stalls shutdown

Looking around to get closer to the root cause

The problem seems to be the script running at shutdown.

I identified the corresponding file with:

find /etc/systemd -name *unattended*

which gaves me the related systemd script:

/etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

which then told me the script executed on shutdown:

/usr/share/unattended-upgrades/unattended-upgrade-shutdown

Investigating deeper to find the root cause

within this script there is a section in line 120 related to the section in /etc/apt/apt.conf.d/50unattended-upgrades -> Unattended-Upgrade::InstallOnShutdown

Line 120 of /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):

The problem: it expects the keyword "False" while in the apt conf we should add "false" (exact string comparison)!

Solution

I was able to fix/workaround the stalling shutdown in 3 different ways:

Workaround A

write "False" instead of "false" in /etc/apt/apt.conf.d/50unattended-upgrades

This setting is upgrade safe until a real fix is provided because the file we change here gets not overwritten by an update of unattended-upgrades. Problem: When the root cause gets fixed this will result in a stalling shutdown again so I suggest to combine this with Workaround B.

OR: Workaround B

decrease the wait time in /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service from default to 15 seconds:

vim /etc/systemd/system/shutdown.target.wants/unattended-upgrades.service

[Service]
Type=oneshot
ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStartSec=15

This setting is NOT upgrade safe because the file we change here may get overwritten by an update of unattended-upgrades. Besides this it is really far away from fixing something but it will ensure that your system will not wait several minutes when shutting down. Keep in mind that after an upgrade of unattended-upgrades you may have to set this again!

OR: Fix C (have to be reported upstream)

fix /usr/share/unattended-upgrades/unattended-upgrades-shutdown to expect "false" instead of "False"

patching /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

--- /tmp/unattended-upgrade-shutdown    2017-02-03 14:53:03.238103238 +0100
+++ /tmp/unattended-upgrade-shutdown_fix    2017-02-03 14:53:17.685589001 +0100
@@ -117,7 +117,7 @@
     # run it
     p = None
     apt_pkg.init_config()
-    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):
+    if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", false):
         env = copy.copy(os.environ)
         env["UNATTENDED_UPGRADES_FORCE_INSTALL_ON_SHUTDOWN"] = "1"
         logging.debug("starting unattended-upgrades in shutdown mode")

Conclusion

tbh only the last one is a real fix. the both other options are just workarounds until the real fix would be implemented.

This has to be done upstream and as this affects both Debian (tested on Debian Stretch) and Ubuntu (tested on Ubuntu 16.04.1) for both distributions.

I have opened a bug report here: https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611