Deploying filesystem images onto a partition of arbitrary size

bootdddeploymentpartimagepartition

I would like to establish a deployment process for Ubuntu machines based on root FS images. I would like to restore the image onto a simply-formatted hard drive (I can completely erase the disk during deployment).

I'd rather keep the config simple and not partition into separate / and /home

So far I've tried using Clonezilla and succeeded in creating an image of the root partition that I can restore. However, using that process leas to the fact that the root partition I restore onto is small, as small as my base image when uncompressed – around 20 gigs. I would like to grow the partition on the actual hard drive to maximum capacity instead, and then restore a filesystem onto it including all symlinks and other bells and whistles.

Also I am looking for a process that would reliably install and configure the bootloader and the boot partition on the drive when the partition is restored onto it. Where should I look? It seems that any dd-based processes are per block, and you are sort of forced to have your destination partition be of the same size as the image. At the same time I cannot guarantee that all the disks of the machines I am going to be deploying this way are of the same capacity… What would be the approach to take? Maybe I need a higher level imaging solution – one that backs up a filesystem instead of a partition?

UPDATE: After doing some prodding into the possible configurations my question comes down to this. How do I establish a pipeline where I could do the following in one step:

  • boot a machine off a DVD/CD
  • format it's hard drive as MBR/root/swap, without any empty space
  • make sure that it's bootable
  • clone the root partition off a network image

So far we've settled on the following workflow:

  • make an image of a small disk (70 gigs)
  • restore it as a whole
  • manually go into gparted and resize the partitions so that the root partition is bigger than it's imaged version

but it has us booting two disks in a row, which is very annoying.

Best Answer

File system restore - while I'm not sure how Bootzilla works, you basically have three options:

  1. Operate on the underlying block device level (i.e. under the file system) - use cloning (e.g. dd) which creates exact copy of the original image. No matter how big partition you have you'll end up with file system of the size of original (there are very likely to be nasty problems if the partition or even physical device is smaller than the original file system).

    This is the fastest option you have (basically limited only by the speed of the source and target devices speeds and their interconnect). On the other hand it's the least flexible one - if you want to have differing sizes of partitions in the end, you have to grow them by hand (or script it). Additionally, by keeping image of the whole block device you are wasting space (the image contains the unused space, although that might be zeroed and the image then compressed).

    If you are maintaining a large number of (mostly) identical systems, this is the way to go since it is the least complicated.

  2. Operate in the file system level - use tools coming with the file system you are using to archive its content and restore it into freshly formatted file system. For example in the case of XFS, you'd use the xfsdump + mkfs.xfs + xfsrestore combo (see man pages for details).

    It is a bit slower but the image you keep contains just the data needed (no wasted space). Which in turn can easily make up for the additional overhead (updating file system structures on restore) if the file system in question is mostly empty. It also gives you the opportunity to create and deploy incremental snapshots. The additional step of creating a new file system allows you to use whatever partition size you need (that is large enough) and tuning each target file system for specific needs (changing block size, meta data space and such). Disadvantage is that it is file system specific.

  3. Operate on the file system level - by using standard tools like tar or cpio (or even rsync or cp)and optionally compression (as in previous cases). You can easily change underlying file systems and the process is pretty straightforward. Extreme attention has to be paid to handling of special files (sparse files, devices/named pipes), since behaviour of the tools varies quite a lot. You also have to make sure you don't cross file system boundary.

Boot loader - don't bother with a separate /boot partition - instead I would strongly suggest considering separate /home. There aren't many cases where having a separate partition for data is unnecessary or even a bad idea.

Depending on the bootloader used, it may even work after full disk clone - first option between disks (not just partitions!) of identical sizes, but generally "reinstalling" should be preferred (and quite easy) - you are copying all necessary configuration files so just running the boot loader installer should be enough.

The only thing one has to be careful about is from where the boot loader reads the data and where does it write - good idea might be chrooting to the new filesystem and running the bootloader from there. Make sure you bind-mount /dev, /proc and possibly /sys from the currently active root file system into the chroot (i.e. the freshly cloned file system).

Note: 20GB partition is quite big - e.g. general desktop Linux installations are usually well under 10GB, and that already includes quite a lot of software.

If your users are installing additional software and/or you (as administrator) are installing it for them (on a per machine basis) it should probably go somewhere else than /usr. Users' software is usually located directly in their homes (which can be quite useful especially for network-mounted home directories), system-specific programs should go into /usr/local, which can easily be another partition. Caches/logs are better kept separately as well. Apart from other things, the clean separation allows to keep the access rights sane - not every user should need to have administrator access to his machine (yes, there are many exceptions to that, but it should be always considered non-standard first). Re-imaging the machine doesn't necessarily destroy everything -- or, put another way -- one can re-image just some parts. And MKFS DEVICE_OF(/var/chaches) is faster than rm -Rf /var/caches. Taking snapshots of the cache data (if necessary) is also easier from a file system.

The downside - disk space fragmentation (into several partitions) should be less of a problem these days due to the average sizes of HDDs.

Related Question