To get closer to a definitive answer for this, we need to see which files are causing the mount: / is busy error. You could do this with:
lsof / | awk 'NR==1 || $4~/[0-9][uw]/'
See my answer to the OP's other question - lsof: show files open as read-write - for the caveats to this. It may be that you need to put this into a separate script and the put the script into the apt hook in order to see something.
My suspicion is that files under /etc remain open once the services are started. Some programs/daemons update their configuration dynamically. NetworkManager and cupsd are two examples. Updates to cups which cause cupsd to scan for new printers (as opposed to a dpkg configuration script) may well be what is causing your issue. I recommend that you put /etc on a writeable file system, even if it is not the source of your problem.
Another possibility is that the filesystem buffer is still in the process of being flushed to disk when you try to do the remount. I'm not sure what the behaviour for mount is here, whether it is to block until IO is complete or to fail and report the disk as busy. The first seems more likely, but I see no sync calls in the output of strace (although possibly the mount system call does this). Anyway, it may work to do a sync before the remount if the lsof above doesn't show anything, eg:
DPkg::Post-Invoke { "sync; mount -o remount /"; };
Bind mount is just... well... a bind mount. I.e. it's not a new mount. It just "links"/"exposes"/"considers" a subdirectory as a new mount point. As such it cannot alter the mount parameters. That's why you're getting complaints:
# mount /mnt/1/lala /mnt/2 -o bind,ro
mount: warning: /mnt/2 seems to be mounted read-write.
But as you said a normal bind mount works:
# mount /mnt/1/lala /mnt/2 -o bind
And then a ro remount also works:
# mount /mnt/1/lala /mnt/2 -o bind,remount,ro
However what happens is that you're changing the whole mount and not just this bind mount. If you take a look at /proc/mounts you'll see that both bind mount and the original mount change to read-only:
/dev/loop0 /mnt/1 ext2 ro,relatime,errors=continue,user_xattr,acl 0 0
/dev/loop0 /mnt/2 ext2 ro,relatime,errors=continue,user_xattr,acl 0 0
So what you're doing is like changing the initial mount to a read-only mount and then doing a bind mount which will of course be read-only.
UPDATE 2016-07-20:
The following are true for 4.5 kernels, but not true for 4.3 kernels (This is wrong. See update #2 below):
The kernel has two flags that control read-only:
- The
MS_READONLY: Indicating whether the mount is read-only
- The
MNT_READONLY: Indicating whether the "user" wants it read-only
On a 4.5 kernel, doing a mount -o bind,ro will actually do the trick. For example, this:
# mkdir /tmp/test
# mkdir /tmp/test/a /tmp/test/b
# mount -t tmpfs none /tmp/test/a
# mkdir /tmp/test/a/d
# mount -o bind,ro /tmp/test/a/d /tmp/test/b
will create a read-only bind mount of /tmp/test/a/d to /tmp/test/b, which will be visible in /proc/mounts as:
none /tmp/test/a tmpfs rw,relatime 0 0
none /tmp/test/b tmpfs ro,relatime 0 0
A more detailed view is visible in /proc/self/mountinfo, which takes into consideration the user view (namespace). The relevant lines will be these:
363 74 0:49 / /tmp/test/a rw,relatime shared:273 - tmpfs none rw
368 74 0:49 /d /tmp/test/b ro,relatime shared:273 - tmpfs none rw
Where on the second line, you can see that it says both ro (MNT_READONLY) and rw (!MS_READONLY).
The end result is this:
# echo a > /tmp/test/a/d/f
# echo a > /tmp/test/b/f
-su: /tmp/test/b/f: Read-only file system
UPDATE 2016-07-20 #2:
A bit more digging into this shows that the behavior in fact depends on the version of libmount which is part of util-linux. Support for this was added with this commit and was released with version 2.27:
commit 9ac77b8a78452eab0612523d27fee52159f5016a
Author: Karel Zak
Date: Mon Aug 17 11:54:26 2015 +0200
libmount: add support for "bind,ro"
Now it's necessary t use two mount(8) calls to create a read-only
mount:
mount /foo /bar -o bind
mount /bar -o remount,ro,bind
This patch allows to specify "bind,ro" and the remount is done
automatically by libmount by additional mount(2) syscall. It's not
atomic of course.
Signed-off-by: Karel Zak
which also provides the workaround. The behavior can be seen using strace on an older and a newer mount:
Old:
mount("/tmp/test/a/d", "/tmp/test/b", 0x222e240, MS_MGC_VAL|MS_RDONLY|MS_BIND, NULL) = 0 <0.000681>
New:
mount("/tmp/test/a/d", "/tmp/test/b", 0x1a8ee90, MS_MGC_VAL|MS_RDONLY|MS_BIND, NULL) = 0 <0.011492>
mount("none", "/tmp/test/b", NULL, MS_RDONLY|MS_REMOUNT|MS_BIND, NULL) = 0 <0.006281>
Conclusion:
To achieve the desired result one needs to run two commands (as @Thomas already said):
mount SRC DST -o bind
mount DST -o remount,ro,bind
Newer versions of mount (util-linux >=2.27) do this automatically when one runs
mount SRC DST -o bind,ro
Best Answer
How can I list those files that prevent / from being remounted to read-only?
A)
fusercan be found in thepsmiscpackage; this is a use case where I findfusershines & is more useful thanlsof.# fuser -v -m / 2>&1 | grep '[Ff]r.e'That will show all processes that have files open on / for reading (f) and writing (F). The files that would prevent / from being remounted to read-only are those that are opened for writing (F).
Kill the processes that are an executable being run with root directory files open for writing., i.e.
# for fupid in $(fuser -v -m / 2>&1 | grep Fr.e | awk '{print $2}'); do kill $fupid; doneThat is above the
systemdcomments with a caveat. Ifsystemdisinitthenfuserwill see it and there are other considerations. Withsystemdrunning, it can (re)start processes behind your back, even if they've just been identified and killed withfuser.systemdis much more advanced than the traditionalsysvinit.B) The UPDATE in the description states the system only has ...
initandgettystill running ...I see the comment that says the system is not using
systemd, it's usinginit. On stretch,systemdisinit. The comment didn't explicitly saysysvinit, so I'm assuming the system in question may be using the default stretchsystemdforinit. Or that other people who stumble on this post, that are using stretch'ssystemd, find this part useful.Per the Debian Wiki,
With
systemdrunning, there are a few additional steps that should be taken to free up / so that it can be remounted without issue.It's likely
system.sliceis holding open files forsystemd-journald.serviceorsystemd-udevd.service(both of which have socket dependencies). Or, ifNetworkManageris running it can respawndhclientwhich writes leases to /var/... (& /var/ isn't always its own device), etc.fusermight find & you killdhclientbutNetworkManagerstarts it right back up.The moral is lots of things are automated that could 'want' / (and even more so with
systemd).To be sure, if it's feasible, the
systemdequivalent of run level 1 is matched byrescue.target(andrunlevel1.targetis a symbolic link torescue.target).1) Start by isolating the system to
rescue.target# systemctl isolate rescue.targetIt should prompt you to enter the root password; follow on screen instructions.
2) At the rescue shell, find out what wants /.
# systemctl show -p Wants /Typically, it's
system.slice; stop everything that Wants /. e.g.# systemctl stop system.slice3) At this point, the remount should not report
mount: / is busyandmount -o remount,ro /should work. If not, check again withfuser.4) FWIW; I've also seen times when
umountfails when/if another device is mounted on a sub-directory of another mount, i.e. nested mounts. For example,umount /would fail if /var/ or /boot/ is on another device (and mounted). Thoughmount -o remount,ro /should still work in this case.lsblkcan be helpful to visualize nested mounts.Why does lsof +L1 no longer list open files that have been unlinked ?
Because they aren't available (sockets or most FIFOs & pipes), they're not open files anymore (the parent process closed the file descriptor), or they (still) have a link count greater than 1.
man lsof(8) details ...