Debian – Unable to Remount Filesystem as Read-Only After Upgrade

aptdebianreadonly

On my Debian system, I have my / on separate partition and mounted read-only. Only /home/, /var/ and /tmp/ are writable. I have also created a Pre-Invoke and Post-Invoke apt hook, so that apt can automatically remount the system for write, when packages are installed or upgraded, and remount it back to read-only when finished:

DPkg::Pre-Invoke  {"mount -o remount,rw / ;};
DPkg::Post-Invoke {"mount -o remount    / ;};

This whole setup works well with one exception. Sometimes during the installation/upgrade process some services need to be restarted or new files opened during the short window when my / is mounted read-write, these files are opened with write permissions. After the installation/upgrade is complete, my Post-Invoke hook returns error because it cannot remount / back to read-only.

Is there some way to solve this problem? This is very annoying, because in this situation I usually must restart the server, which is impractical.

EDIT

Below is a log of my latest package upgrade, which resulted in the described error:

root@alpha# apt-get upgrade 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  base-files curl libc-bin libc6 libcurl3 libcurl3-gnutls libmysqlclient18 libssl1.0.0 locales multiarch-support mysql-client mysql-client-5.5 mysql-common
  nscd openssl tzdata wget whois
18 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 18.7 MB of archives.
After this operation, 264 kB of additional disk space will be used.
Do you want to continue [Y/n]? 
Fetched 18.7 MB in 0s (33.2 MB/s) 
Preconfiguring packages ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace base-files 7.1wheezy3 (using .../base-files_7.1wheezy4_amd64.deb) ...
Unpacking replacement base-files ...
Processing triggers for man-db ...
Processing triggers for install-info ...
Setting up base-files (7.1wheezy4) ...
Installing new version of config file /etc/debian_version ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace libc-bin 2.13-38 (using .../libc-bin_2.13-38+deb7u1_amd64.deb) ...
Unpacking replacement libc-bin ...
Processing triggers for man-db ...
Setting up libc-bin (2.13-38+deb7u1) ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace libc6:amd64 2.13-38 (using .../libc6_2.13-38+deb7u1_amd64.deb) ...
Unpacking replacement libc6:amd64 ...
Setting up libc6:amd64 (2.13-38+deb7u1) ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace libssl1.0.0:amd64 1.0.1e-2+deb7u1 (using .../libssl1.0.0_1.0.1e-2+deb7u4_amd64.deb) ...
Unpacking replacement libssl1.0.0:amd64 ...
Preparing to replace curl 7.26.0-1+wheezy7 (using .../curl_7.26.0-1+wheezy8_amd64.deb) ...
Unpacking replacement curl ...
Preparing to replace libcurl3:amd64 7.26.0-1+wheezy7 (using .../libcurl3_7.26.0-1+wheezy8_amd64.deb) ...
Unpacking replacement libcurl3:amd64 ...
Preparing to replace libcurl3-gnutls:amd64 7.26.0-1+wheezy7 (using .../libcurl3-gnutls_7.26.0-1+wheezy8_amd64.deb) ...
Unpacking replacement libcurl3-gnutls:amd64 ...
Preparing to replace mysql-common 5.5.33+dfsg-0+wheezy1 (using .../mysql-common_5.5.35+dfsg-0+wheezy1_all.deb) ...
Unpacking replacement mysql-common ...
Preparing to replace libmysqlclient18:amd64 5.5.33+dfsg-0+wheezy1 (using .../libmysqlclient18_5.5.35+dfsg-0+wheezy1_amd64.deb) ...
Unpacking replacement libmysqlclient18:amd64 ...
Preparing to replace multiarch-support 2.13-38 (using .../multiarch-support_2.13-38+deb7u1_amd64.deb) ...
Unpacking replacement multiarch-support ...
Processing triggers for man-db ...
Setting up multiarch-support (2.13-38+deb7u1) ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace tzdata 2013h-0wheezy1 (using .../tzdata_2013i-0wheezy1_all.deb) ...
Unpacking replacement tzdata ...
Setting up tzdata (2013i-0wheezy1) ...

Current default time zone: 'Europe/London'
Local time is now:      Sat Feb 15 11:35:41 CET 2014.
Universal Time is now:  Sat Feb 15 11:35:41 UTC 2014.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

(Reading database ... 20511 files and directories currently installed.)
Preparing to replace wget 1.13.4-3 (using .../wget_1.13.4-3+deb7u1_amd64.deb) ...
Unpacking replacement wget ...
Preparing to replace locales 2.13-38 (using .../locales_2.13-38+deb7u1_all.deb) ...
Unpacking replacement locales ...
Preparing to replace whois 5.0.23 (using .../whois_5.1.1~deb7u1_amd64.deb) ...
Unpacking replacement whois ...
Preparing to replace mysql-client 5.5.33+dfsg-0+wheezy1 (using .../mysql-client_5.5.35+dfsg-0+wheezy1_all.deb) ...
Unpacking replacement mysql-client ...
Preparing to replace mysql-client-5.5 5.5.33+dfsg-0+wheezy1 (using .../mysql-client-5.5_5.5.35+dfsg-0+wheezy1_amd64.deb) ...
Unpacking replacement mysql-client-5.5 ...
Preparing to replace nscd 2.13-38 (using .../nscd_2.13-38+deb7u1_amd64.deb) ...
[ ok ] Stopping Name Service Cache Daemon: nscd.
Unpacking replacement nscd ...
Preparing to replace openssl 1.0.1e-2+deb7u1 (using .../openssl_1.0.1e-2+deb7u4_amd64.deb) ...
Unpacking replacement openssl ...
Processing triggers for install-info ...
Processing triggers for man-db ...
Setting up libssl1.0.0:amd64 (1.0.1e-2+deb7u4) ...
Setting up libcurl3:amd64 (7.26.0-1+wheezy8) ...
Setting up curl (7.26.0-1+wheezy8) ...
Setting up libcurl3-gnutls:amd64 (7.26.0-1+wheezy8) ...
Setting up mysql-common (5.5.35+dfsg-0+wheezy1) ...
Setting up libmysqlclient18:amd64 (5.5.35+dfsg-0+wheezy1) ...
Setting up wget (1.13.4-3+deb7u1) ...
Setting up locales (2.13-38+deb7u1) ...
Generating locales (this might take a while)...
  en_DK.UTF-8... done
  en_US.UTF-8... done
Generation complete.
Setting up whois (5.1.1~deb7u1) ...
Setting up mysql-client-5.5 (5.5.35+dfsg-0+wheezy1) ...
Setting up mysql-client (5.5.35+dfsg-0+wheezy1) ...
Setting up nscd (2.13-38+deb7u1) ...
[ ok ] Starting Name Service Cache Daemon: nscd.
Setting up openssl (1.0.1e-2+deb7u4) ...
mount: / is busy

The last line (mount: / is busy) is the error returned by apt, when trying to remount / back to read-only.

UPDATE:

the command suggested by Graeme does not show any files:

# lsof / | awk 'NR==1 || $4~/[0-9][uw]/'
COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME

Best Answer

To get closer to a definitive answer for this, we need to see which files are causing the mount: / is busy error. You could do this with:

lsof / | awk 'NR==1 || $4~/[0-9][uw]/'

See my answer to the OP's other question - lsof: show files open as read-write - for the caveats to this. It may be that you need to put this into a separate script and the put the script into the apt hook in order to see something.

My suspicion is that files under /etc remain open once the services are started. Some programs/daemons update their configuration dynamically. NetworkManager and cupsd are two examples. Updates to cups which cause cupsd to scan for new printers (as opposed to a dpkg configuration script) may well be what is causing your issue. I recommend that you put /etc on a writeable file system, even if it is not the source of your problem.

Another possibility is that the filesystem buffer is still in the process of being flushed to disk when you try to do the remount. I'm not sure what the behaviour for mount is here, whether it is to block until IO is complete or to fail and report the disk as busy. The first seems more likely, but I see no sync calls in the output of strace (although possibly the mount system call does this). Anyway, it may work to do a sync before the remount if the lsof above doesn't show anything, eg:

DPkg::Post-Invoke { "sync; mount -o remount /"; };
Related Question