Linux Ext4 – Force Root Filesystem Check at Boot

ext4fscklinux

Yesterday, one of our computers dropped to grub shell or honestly, I am unsure what shell it was when we turned on the machine.

It showed that it can't mount the root filesystem or something in this sense, because of inconsistencies.

I ran, I believe:

fsck -fy /dev/sda2

Rebooted and the problem was gone.

Here comes the question part:

I already have in her root's crontab:

@reboot /home/ruzena/Development/bash/fs-check.sh

while the script contains:

#!/bin/bash
touch /forcefsck

Thinking about it, I don't know, why I created script file for such a short command, but anyways…

Further, in the file:

/etc/default/rcS

I have defined:

FSCKFIX=yes

So I don't get it. How could the situation even arise?

What should I do to force the root filesystem check (and optionally a fix) at boot?

Or are these two things the maximum, that I can do?

OS: Linux Mint 18.x Cinnamon 64-bit.

fstab:

cat /etc/fstab | grep ext4

shows:

UUID=a121371e-eb12-43a0-a5ae-11af58ad09f4    /    ext4    errors=remount-ro    0    1

grub:

fsck.mode=force

was already added to the grub configuration.

Best Answer

`ext4` filesystem check during boot

Tested on OS: Linux Mint 18.x in a Virtual Machine

Basic information

/etc/fstab has the fsck order as the last (6th) column, for instance:

<file system>    <mount point>    <type>    <options>    <dump>    <fsck>
UUID=2fbcf5e7-1234-abcd-88e8-a72d15580c99 / ext4 errors=remount-ro 0 1

FSCKFIX=yes variable in /etc/default/rcS

This will change the fsck to auto fix, but not force a fsck check.

From man rcS:

FSCKFIX
    When  the  root  and all other file systems are checked, fsck is
    invoked with the -a option which means "autorepair".   If  there
    are  major  inconsistencies then the fsck process will bail out.
    The system will print a  message  asking  the  administrator  to
    repair  the  file  system manually and will present a root shell
    prompt (actually a sulogin prompt) on the console.  Setting this
    option  to  yes  causes  the fsck commands to be run with the -y
    option instead of the -a option.  This will tell fsck always  to
    repair the file systems without asking for permission.

From man tune2fs

If you are using journaling on your filesystem, your filesystem
will never be marked dirty, so it will not normally be checked.

Start with

Setting the following

FSCKFIX=yes

in the file

/etc/default/rcS

Check and note last time fs was checked:

sudo tune2fs -l /dev/sda1 | grep "Last checked"

These two options did NOT work

Passing -F (force fsck on reboot) argument to shutdown:
```
shutdown -rF now
```
Nope; see: man shutdown.

Adding the /forcefsck empty file with:

touch /forcefsck

These scripts seem to use this:

/etc/init.d/checkfs.sh
/etc/init.d/checkroot.sh

did NOT work on reboot, but the file was deleted.

Verified by:

sudo tune2fs -l /dev/sda1 | grep "Last checked"
sudo less /var/log/fsck/checkfs
sudo less /var/log/fsck/checkroot

These seem to be the logs for the init scripts.

I repeat, these two options did NOT work!

Both of these methods DID work

systemd-fsck kernel boot switches

Editing the main grub configuration file:
```
sudoedit /etc/default/grub
```
```
GRUB_CMDLINE_LINUX="fsck.mode=force"
```
```
sudo update-grub
sudo reboot
```
This did do a file system check as verified with:
```
sudo tune2fs -l /dev/sda1 | grep "Last checked"
```
Note: This DID a check, but to force a fix too, you need to specify fsck.repair="preen", or fsck.repair="yes".
Using tune2fs to set the number of file system mounts before doing a fsck, man tune2fs
```
tune2fs' info is kept in the file system superblock
```
-c switch sets the number of times to mount the fs before checking the fs.
```
sudo tune2fs -c 1 /dev/sda1
```
Verify with:
```
sudo tune2fs -l /dev/sda1
```
This DID work as verified with:
```
sudo tune2fs -l /dev/sda1 | grep "Last checked"
```

Summary

To force a fsck on every boot on Linux Mint 18.x, use either tune2fs, or fsck.mode=force, with optional fsck.repair=preen / fsck.repair=yes, the kernel command line switches.

Related Solutions

How to avoid “Run fsck manually” messages while allowing experimenting with system time changes

I was going to suggest hacking e2fsck to disable the specific checks for a last mount time or last write times in the future. These are defined in problem.c / problem.h, and used in super.c. But in looking, I discovered that E2fsprogs 1.41.10 adds a new option to /etc/e2fsck.conf called broken_system_clock. This seems to be exactly what you need, and since you're using Red Hat Enterprise Linux 6, you should have 1.41.12, which includes this option. From the man page:

   broken_system_clock
          The e2fsck(8) program has some hueristics that assume  that  the
          system clock is correct.  In addition, many system programs make
          similar assumptions.  For example, the UUID library  depends  on
          time  not going backwards in order for it to be able to make its
          guarantees about issuing universally unique ID’s.  Systems  with
          broken  system clocks, are well, broken.  However, broken system
          clocks, particularly in embedded systems, do exist.  E2fsck will
          attempt  to  use  hueristics to determine if the time can no tbe
          trusted; and to skip time-based checks if this is true.  If this
          boolean  is set to true, then e2fsck will always assume that the
          system clock can not be trusted.

Yes, the man page can't spell "heuristics". Oops. But presumably the code works anyway. :)

Ubuntu – ext4 filesystems frequently corrupting

All three data journaling modes should leave the filesystem itself fully intact after a power failure. So it should always mount without errors. The difference is only in the data in your files; data=writeback mode may leave stale data (i.e., what was stored in the disk sectors before the writes your app did). data=ordered and data=journaled should not do this.

Most likely what you're seeing is that I/O barriers aren't working on your setup. First, make sure you're not mounting with barrier=0/nobarrier. That boosts performance, but will cause corruption on power failure.

If I/O barriers are on, it's also possible you're passing through a storage layer that doesn't support them. On older releases, LVM didn't and various mdraid levels didn't. (This was fixed in Linux 2.6.33; so only if you're running Lucid still.)

Finally, it's possible your disks are telling lies. Disks have write caches. Especially with NCQ, they're supposed to only tell the OS they've written data when they've actually done so, but they've been known to tell the OS its written when its only in the disk's write cache. Increases performance. At least as long as the power stays on. You can try disabling the write cache on the disks, though you'll take a performance hit for this.

Note also that flash-memory disks have a lot of work to do under the hood, and many of them don't handle power failure well. (For example, wear leveling sometimes requires that a full flash block of data be moved. If the power fails in the middle, bad things happen on some flash disks.)

Finally... have you considered an UPS?