Systemd – Check If Device Is Being Unmounted and Why

fstabmountsystemdsystemd-mount

This is a followup to another question.

I figured out something is unmounting my device right after I mount it.

This device is being used by a database (Vertica), which is down and not using the directory while I'm running the mount command.

I'm trying to figure out:

  1. Is systemd the one which unmounts the device?
  2. How can I debug why is that happening?
  3. How do I fix it?

Here's an example of what is happening:

[root@mymachine systemd]# mount -t ext4 /dev/xvdx /vols/data5; ls -la /vols/data5; sleep 5; ls -la /vols/data5
total 36
drwxr-xr-x   5 dbadmin verticadba  4096 Jul 23  2017 .
drwxr-xr-x   9 root    root          96 Jul 16 18:52 ..
drwxrwx--- 503 dbadmin verticadba 12288 Jul 23 13:51 somedb
drwx------   2 root    root       16384 Nov 30  2016 lost+found
drwxrwxrwx   2 dbadmin verticadba  4096 Jun 20 08:32 tmp
total 0
drwxr-xr-x 2 root root  6 Jun  8  2017 .
drwxr-xr-x 9 root root 96 Jul 16 18:52 ..
[root@mymachine ~]# 

fstab:

#
# /etc/fstab
# Created by anaconda on Mon May  1 18:59:01 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=29342a0b-e20f-4676-9ecf-dfdf02ef6683 /                       xfs     defaults        0 0
/dev/xvdb swap swap defaults,nofail 0 0
/dev/xvdy /vols/data ext4 defaults 0 0
/dev/xvdx /vols/data5 ext4 defaults 0 0

Some more logs as per Filipe Brandenburger's suggestion:

Aug 01 16:55:19 mymachine kernel: EXT4-fs (xvdx): mounted filesystem with ordered data mode. Opts: (null)
Aug 01 16:55:19 mymachine systemd[1]: Unit vols-data5.mount is bound to inactive unit dev-xvdl.device. Stopping, too.
Aug 01 16:55:19 mymachine systemd[1]: Unmounting /vols/data5...
Aug 01 16:55:19 mymachine umount[353194]: umount: /vols/data5: target is busy.
Aug 01 16:55:19 mymachine umount[353194]: (In some cases useful info about processes that use
Aug 01 16:55:19 mymachine umount[353194]: the device is found by lsof(8) or fuser(1))
Aug 01 16:55:19 mymachine systemd[1]: vols-data5.mount mount process exited, code=exited status=32
Aug 01 16:55:19 mymachine systemd[1]: Failed unmounting /vols/data5.

Best Answer

Ok, that was in interesting debugging experience... Thanks Filipe Brandenburger for leading me to it!

  1. Is systemd the one which unmounts the device?

Yes. journalctl -e shows a related message:

Aug 01 16:55:19 mymachine systemd[1]: Unit vols-data5.mount is bound to inactive unit dev-xvdl.device. Stopping, too.

Apparently I'm not the first one to encounter it. See this systemd issue: systemd umounts manual mounts when it has a failed unit for that mount point #1741

  1. How can I debug why is that happening?

Run journalctl -e for debugging.

  1. How do I fix it?

This workaround worked for me: run the command below, then try mounting again.

systemctl daemon-reload

That's all, folks!

Related Question