Linux – Interactively terminate “A stop job is running” at shutdown

arch linuxshutdownsystemd

Usually when I shut down my machine (with the shutdown command) it works pretty quickly, but every once in a while, usually after a borked update or when I break some service's config file I end up with

A stop job is running for Some Broken Service (20min 58s / no limit)

parading across my screen until I give in and hold down the power button until the machine turns off. Sometimes there are multiple jobs, with one actually hung and the others waiting on the hung one, with a 1:30 timeout per job adding up to 20-30 minutes. Sometimes it will eventually turn off after a lot of waiting, sometimes it won't (or I run out of patience).

Is there anything I can do when I discover this to forcibly terminate whatever's hung and allow it to continue shutting down (semi)-gracefully? Mainly, I want to make sure btrfs always has an opportunity to remount-ro and flush-to-disk because I'm running (semi-unstable) raid6. I've heard Ctrl+Alt+Delete should help, but all it seems to do is print ^[[3~ at the bottom of my screen — is there something I need to do to enable it?

To be clear, I'm not trying to globally turn down or disable this timeout — my system is usually able to shut down without running in to it — I just want a way to bypass it interactively when I discover something's wrong. Also, not interested in why this happens or how to solve it — diagnosing that has to wait until the machine finishes shutting down and starting back up.


The most recent time I ran into this was by writing a udev rule that caused a kernel panic, causing systemd-udev to be permanently hung as far as I can tell. For some reason, that meant docker hung for 30 minutes until the shutdown itself timed out.

It also seems to happen frequently when I have a systemd .mount unit pointing at a NFS share only accessibly over a VPN, when the VPN has died. Stopping the mount unit will then hang forever, causing for some reason my user session and 6-8 other units to hang for 1:30 each (in sequence) until they all time out.

Also used to happen a while back every time I shut down for some inexplicable reason, then was fixed by a kernel upgrade (5.7.something fixed it, if I recall correctly).

Best Answer

Is there anything I can do when I discover this to forcibly terminate whatever's hung and allow it to continue shutting down (semi)-gracefully?

SysRq + REISUB has been 100% safe in my experience. To be honest, I even use a shorter version SysRq + SUB.


Note: Most SysRq functions are disabled by default on most distributions. To re-enable them, create the file /etc/sysctl.d/90-sysrq.conf containing

 kernel.sysrq = 1

This will take affect next boot, but to enable it immediately run

 sysctl --system
Related Question