Linux – Graceful shutdown with suspend job hanging in syscall

freezeiokernellinuxsuspend

When I suspended the system yesterday the job didn’t terminate
and I’ve got a systemd-suspend.service job hanging in
uninterruptible sleep ever since:

# systemctl list-jobs
  JOB UNIT                    TYPE  STATE  
21595 post-resume.target      start waiting
21593 systemd-suspend.service start running
21592 suspend.target          start waiting
21596 post-resume.service     start waiting

# systemctl status systemd-suspend.service
● systemd-suspend.service - Suspend
   Loaded: loaded (/nix/store/2jspk70lir7jcn1krax8haw2j7486i3a-systemd-243.3/example/systemd/system/systemd-suspend.se>
   Active: activating (start) since Sat 2020-04-04 03:07:36 CEST; 23h ago
     Docs: man:systemd-suspend.service(8)
 Main PID: 16761 (systemd-sleep)
       IP: 0B in, 0B out
    Tasks: 1 (limit: 4915)
   Memory: 1.0M
      CPU: 20ms
   CGroup: /system.slice/systemd-suspend.service
           └─16761 /nix/store/2jspk70lir7jcn1krax8haw2j7486i3a-systemd-243.3/lib/systemd/systemd-sleep suspend

Apr 04 03:07:36 phlegethon systemd[1]: Starting Suspend...
Apr 04 03:07:36 phlegethon systemd-sleep[16761]: Suspending system...

# ps aux |grep suspend
root     16761  0.0  0.0  10364  2052 ?        Ds   Apr04   0:00 /nix/store/2jspk70lir7jcn1krax8haw2j7486i3a-systemd-243.3/lib/systemd/systemd-sleep suspend

When I try to trigger suspend manually, the kernel (5.4.14)
answers EBUSY:

# echo mem >/sys/power/state
-bash: echo: write error: Device or resource busy

Looks like the kernel is stuck syncing one of the disks:

# cat /proc/16761/stack
[<0>] iterate_bdevs+0x98/0x142
[<0>] ksys_sync+0x6e/0xb0
[<0>] ksys_sync_helper+0x13/0x90
[<0>] pm_suspend.cold.8+0x213/0x361
[<0>] state_store+0x80/0xe0
[<0>] kernfs_fop_write+0xc1/0x1a0
[<0>] vfs_write+0xa5/0x1a0
[<0>] ksys_write+0x59/0xd0
[<0>] do_syscall_64+0x4e/0x120
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Now in this state I can’t even seem to poweroff the machine
normally:

# systemctl poweroff
Failed to power off system via logind: There's already a shutdown or sleep operation in progress

So what do I do? I am tempted to reach for the big gun (Sysrq)
but I wonder whether it will actually work? sync(1) as
expected just hangs, so I am reluctant to try.

Also, is there a way to find out which disk device the kernel
is waiting on? I am kind of hoping it is just some USB device,
nothing serious.

Best Answer

I know this doesn't exactly answer the question but maybe it will help somewhat.
You or someone else who will read this.

I had (or still have) similar problem now. Suspend didn't finish returning to the system with the job hanging.

24913 systemd-suspend.service start running
24912 suspend.target          start waiting

I was getting the same message as you when trying to continue.

# systemctl suspend
Failed to suspend system via logind: There's already a shutdown or sleep operation in progress


My goal was to suspend, not shutdown. Here is what I did.

# systemctl cancel
# systemctl stop systemd-suspend.service

First, to stop the hanging operation.
Second, caused the system to suspend instantly.

After canceling I tried to suspend by systemctl suspend but the result was getting the same problem again.
After waking up I started the service, which has suspended the system again.

Hopefully I'll be able to suspend without this workaround now.

Related Question