Linux – read only root filesystem

filesystemslinuxreadonly

Somehow my Debian went to read only in root file system. I have no idea how this could have happened.
For example when I am in /root folder and type command nano and after that press Tab to list possible file in that folder I get the message:

root@debian:~# nano -bash: cannot create temp file for here-document: Read-only file system

The same for the cd command when I type cd /home and press Tab to list paths I have this:

root@debian:~# cd /home -bash: cannot create temp file for here-document: Read-only file system

I also have problems with software like apt and others. Can't even apt-get update. I have a lot of errors like this:

Err http ://ftp.de.debian.org wheezy-updates/main Sources
406  Not Acceptable
W: Not using locking for read only lock file /var/lib/apt/lists/lock
W: Failed to fetch http ://ftp.de.debian.org/debian/dists/wheezy/Release  rename failed, Read-only file system (/var/lib/apt/lists/ftp.de.debian.org_debian_dists_wheezy_Release -> /var/lib/apt/lists/ftp.de.debian.org_debian_dists_wheezy_Release).
W: Failed to fetch http ://security.debian.org/dists/wheezy/updates/main/source/Sources  404  Not Found
W: Failed to fetch http ://security.debian.org/dists/wheezy/updates/main/binary-amd64/Packages  404  Not Found
W: Failed to fetch http ://ftp.de.debian.org/debian/dists/wheezy-updates/main/source/Sources  406  Not Acceptable
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Not using locking for read only lock file /var/lib/dpkg/lock

I have a lot of problems in the system.
Is it possible to fix that? How can I check what happened? What should I look for in the logs?

I know it could be because of the line in /etc/fstab file:

/dev/mapper/debian-root /               ext4    errors=remount-ro 0       1

but what is the problem? I can't find nothing or maybe I don't know where to look.

Edit:

I did search messages logs and found only this:

kernel: [    5.709326] EXT4-fs (dm-0): re-mounted. Opts: (null)
kernel: [    5.977131] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro
kernel: [    7.174856] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)

I guess it's correct, because I have the same entries on other debian machines.

I found something in dmesg (I cut that output a bit because was a lot standard ext4 things)

root@gs3-svn:/# dmesg |grep ext4
EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4507: Journal has aborted
EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4507: Journal has aborted
EXT4-fs error (device dm-0) in ext4_dirty_inode:4634: Journal has aborted
EXT4-fs error (device dm-0): ext4_discard_preallocations:3894: comm rsyslogd: Error loading buddy information for 1
EXT4-fs warning (device dm-0): ext4_end_bio:250: I/O error -5 writing to inode 133130 (offset 132726784 size 8192 starting block 159380)
EXT4-fs error (device dm-0): ext4_journal_start_sb:327: Detected aborted journal

5 errors and 1 warning. Any ideas? Is it safe to use mount -o remount,rw / ?

Best Answer

The default behaviour for most Linux file systems is to safeguard your data. When the kernel detects an error in the storage subsystem it will make the filesystem read-only to prevent (further) data corruption.

You can tune this somewhat with the mount option errors={continue|remount-ro|panic} which are documented in the system manual (man mount).

When your root file-system encounters such an error, most of the time the error won't be recorded in your log-files, as they will now be read-only too. Fortunately since it is a kernel action the original error message is recorded in memory first, in the kernel ring buffer. Unless already flushed from memory you can display the contents of the ring buffer with the dmesg command. .

Most real hard disks support SMART and you can use smartctl to try and diagnose the disk health.

Depending on the error messages, you could decide it is still safe to use file-system and return it read-write condition with mount -o remount,rw /

In general though, disk errors are a precursor to complete disk failure. Now is the time to create a back-up of your data or to confirm the status of your existing back-ups.

Related Question