Ubuntu – How to recover data from /var on a broken system

The alternative - LVM Snapshots - You can read about LVM snapshots here: http://www.tutonics.com/2012/12/lvm-guide-part-2-snapshots.html

In short - snapshots are a way of freezing the used sectors on the HardDisk, telling the system to write any change/delete/add on a reserved part of the HardDisk, called a snapshot volume, thus postpone the decision of keeping/discarding the changes on the drive to some future time, as far as the snapshot volume space provides. And so, once you decide to rollback the changes (or keep them), you can do it with one line.

It may seem complex at first, but I can assure you that after the first time effort, all that is needed to use the snapshot mechanism is just 2 line scripts (downloads provided!)

Warranty - No warranty what so ever is given for the instructions below. Use at your own risk!

How?

If you don't have a pre-configured volume group, you will probably need to install a fresh clean Ubuntu. Then, you will have to put some effort into configuring the system correctly for the first time, but from this point on, snapshot will become effortless. And by effortless I mean one liner commands, using pre-configured scripts.

1. Boot to the ubuntu live cd (no installation yet.. just the live cd)
2. Open Accessories -> Terminal

3. We will be clearing the hard drive! Please backup anything important before going through this stage.

   First, let's remove the existing partitions:

   # fdisk /dev/sda
     Enter 'd' to delete - enter for as many partitions you have 
   > d   
   > <enter>
     ...
     Enter 'w' to write down changes
   > w
   

4. Create the boot partition and the rest of the drive as the other partition

   # fdisk /dev/sda
   > n
     To create a new partition.
   > p
     For Boot Primary partition.
   > 1
     Press <enter> to accept the default start sector.
     Enter:
   > +512M
     To give the boot partition a size of 512MB
   > n
     To create (another) new partition.
   > p
     Should be Primary too.
   > <enter> 
     To accept the default value of 2 (the second partition)
   > <enter>
     To accept the default start sector
   > <enter>
     To accept the whole disk
   > t
     To change the type of the second partition to LVM
   > 2
     To choose the second partition
   > 8e
     To set the type to 'LVM Linux'
   > w
     To write changes and exit.
   

   To check that the changes worked well run:

   # fdisk --list
   

   You should get something like this:

     Device     Boot   Start    End        Blocks     Id   System
     /dev/sda1         2048     524287     261120     83   Linux
     /dev/sda2         524288   488397167  243936440  8e   Linux LVM
   

5. Remove any previous volume groups / logical volumes:

   # lvdisplay 
   

   Will display the logical volumes you may have on the system

   # lvremove <existing_logical_volume_name>
   

   If any volume install that may interfere with this installation need be removed.

   # vgdisplay 
   

   Will show any existing volume group on the system

   # vgremove <existing_volume_group>
   

   Same as for the volume groups..

6. Create the Volume Group 'vgsnap'

   # vgcreate vgsnap /dev/sda2
   

   Should result with:

     Volume group "vgsnap" successfully created
   

7. Create the Logical Volume

   # lvcreate --extents 100%FREE --name lvroot vgsnap
   

   The root logical volume, starting with 100% of device.

   # lvresize --size -30G /dev/mapper/vgsnap-lvroot
   

   This will take off 30G for swap & snapshot volumes. Of course - you can choose smaller sizes for the swap/snapshot.

   > 'y'
     for the warning shown.
   

   And create the swap volume:

   # lvcreate --size 15G --name lvswap vgsnap
   

   Note: we leave now 15G of unused drive space, reserved for the future snapshot.

8. Installing Ubuntu

   Close the terimnal and start the Installer.

   Choose "Use LVM..." (You may also choose to enable whole disk encryption - but I didn't test it..)

   Select "Something Else" and press continue. At this stage you will find allot of /dev/mapper/... volumes defined. Some of these are auto generated (lvsnap-*) - you can ignore them for now.

   Select: /dev/mapper/vgsnap-lvroot volume. Choose "ext4" as format, and mount as '/'

   Select: /dev/mapper/vgsnap-lvswap volume, and choose "swap area"

   If you are to create an encrypted drive (I have no clue if that would work with the snapshots?), you will probably want to select the /dev/sda2 and mark the "volume for encryption". Didn't test it though..

   Note: The unused 15G are left free for the snapshot volume.

   Select: /dev/sda1 partition. Format it as 'ext2' and mount it as '/boot'

   Below - choose: /dev/sda1 partition as boot

   Install!

Using the snapshot

The following steps will show how to use the snapshot at any given time:

1. Mounting the snapshot:

   Create the snapshot volume

   # lvcreate --size 15G --snapshot --name lvsnap /dev/mapper/vgsnap-lvroot
   

   Create the mount directory

   # mkdir /mnt/vgsnap-lvsnap
   

   Mount the snapshot as read-only mount

   # mount -o ro /dev/vgsnap/lvsnap  /mnt/vgsnap-lvsnap
   

   Snapshot is On!

   At this point you may start messing up your system, installing beta drivers, and even (No! That's just inappropriate).

   From now on you may at any time (as far as the snapshot size provides..) return to previous state, or, if satisfied with the new look - keep it.

2. Automatic snapshot volume free disk space monitoring: Using a tweaked version of the script provided here: http://blog.rimuhosting.com/2012/02/20/cron-script-for-checking-disk-space/ I created monitor_snapshot script.

   How to use/install it? - Pretty much as with the link above:

   # cd /etc/cron.hourly
   # copy /home/<user>/Downloads/monitor_snapshot
   # chmod +x monitor_snapshot
   

   Before running it for the first time you better edit the values in it, such as your email address (where the notifications are to be sent, etc..) If properly configured, the cron daemon will run the script every X hours/days - as you decide - and notify you once the snapshot is about to be full.

   To test the script run:

   # sh /etc/cron.hourly/monitor_snapshot
   

   It will print the percentage (number) of free space in the snapshot.

   Important! Once you're done with your tests/installations - rollback or keep changes - and Umount the snapshot!

3. Rolling back changes: (See rollback_snapshot script, with the following instructions in it, link at the end of document)

   # lvconvert --merge /dev/vgsnap/lvsnap
   

   This will also umount the snapshot, but will discard any of your changes. Also - You will see a warning that suggest that the revert will only occur next activation, so:

   # reboot
   

   Note: If you skip merging in, your changes shall be committed once you Umount the snapshot.

4. Committing changes - umounting the snapshot: (See commit_snapshot script, with the following instructions in it - link at the end of document)

   # umount /mnt/vgsnap-lvsnap
     Will unmount (commit0 the snapshot
   
   # lvremove /dev/mapper/vgsnap-lvsnap
   # rm -rf /mnt/vgsnap-lvsnap/
   
   # rm /etc/cron.hourly/monitor_snapshot
     Remove the cron daemon script (that is unused)
   

5. Re-mounting the snapshot: (See remount script, with the following instructions in it - link at the end of document) Pretty much as before:

   # lvcreate --size 15G --snapshot --name lvsnap /dev/mapper/vgsnap-lvroot
   # mkdir /mnt/vgsnap-lvsnap
   # mount -o ro /dev/vgsnap/lvsnap /mnt/vgsnap-lvsnap
   

   If you saved the monitor_snapshot at your home dir you can restore it back:

   # cp /home/<user>/monitor_snapshot /etc/cron.hourly/
   # chmod +x /etc/cron.hourly/monitor_snapshot
   

That's all folks! Following instructions are for quitters only ;)

1. Discarding the snapshot volume (If no longer required)

   # rmdir /mnt/vgsnap-lvsnap
   # lvremove /dev/vgsnap/lvsnap
   

   (You might also want to resize the root partition or swap to occupy the missing 15G reserved for swap)

2. Discarding the volume group - Possible only by removing the volume group's volumes, which probably means deleting the OS installation. But of course, there's no real need to remove the volume group...

   To remove the volume group:

   # vgremove vgsnap
   

Important Note! - formatting / running fdisk and removing all partitions will NOT remove the volumes nor the group, as these are written on a different part of the drive... So, just use the remove instructions given above and remove the volumes and group if required.

After a discussion with OP, a working solution was found. Just boot into a live system, open a terminal and type:

sudo e2fsck -f -C 0 /dev/sdxy

where sdxy is the corresponding partition. This should solve the issue.

Note : e2fsck works on Linux ext2/ext3/ext4 file systems. If you have a different file system, execute sudo fsck /dev/sdxy -f instead.