Critique My Backup and Storage Plan

backupstorage

My current storage (RAID-1 off of a hardware RAID card) and backup (a spare drive) solutions for my home network are inadequate. I have too much data scattered on various one-off drives. It is time to evolve. Backups seem simple enough, at least: lots of big drives. However, I am bewildered by the number of choices for small home storage. The Drobo S looks appealing. So does the ReadyNAS. I am not looking for bunches of shiny features, I'm mostly interested in reliability. I am not interested in building Yet Another PC to create a file server or doing something in the cloud, or whatever. I'm stupid, so I am keeping it simple.

Requirements for Main Volume:

  1. Starting working space roughly 2TB, with options for growth up to 5TB
  2. RAID or something RAID-like with at
    least one parity drive
  3. eSATA II for speed during backups
  4. Ability to shut down gracefully when
    alerted of low power by a UPS
  5. Optional but Desirable: Will take
    2TB drives now with options for the
    larger 3TB drives coming in
    2010-2011
  6. Optional but Desirable: : RAID-6 or
    something similar, with two parity
    drives
  7. Optional but Desirable: : Hot spare
  8. Ethernet connection not required, as
    the volume will be shared via the
    same machines which runs my home
    print server

Backups:

  1. Backup performed via ROBOCOPY in
    mirror mode to an external hard
    drive via a eSATA II connection.
  2. Start with rotating between two
    external 2TB hard drives, will go up
    to six external 2TB drives.
  3. Start with a weekly backup, move to
    a bi-weekly backup as more drives
    are added.
  4. Move to 3TB drives as the size of my
    main volume increases.
  5. Backup drives will be stored on an
    off-site location.

Hard drives:

  1. I plan on buying all of the same
    model, but different batches from
    different vendors.
  2. I found a "burn-in" utility with
    which I can pound away on the drives
    for a couple of weeks before adding
    them to the backup pool or the main
    volume.

I estimate that I am looking at roughly $1,500 to start, once I start throwing in two TB drives for backup and four for storage. So, are there any obvious flaws in my plan? What have I overlooked? Any suggestions for the storage device for my main volume that fits my requirements? Or do I just keep it simple, 2 drives in RAID-1, then perform due diligence with my backups, accepting that I will have to buy a whole new unit when my data grows past 2TB?

Best Answer

Main Volume

I know you said you don't want to build another PC fileserver, but most of the ready-to-go solutions don't have any safeguards against silent data corruption.

If you're looking for data integrity and reliability, you might want to consider running an OpenSolaris fileserver with a raidz2 or raidz3 configuration (2 or 3 parity drives, respectively) on ZFS.

With larger drives, the rebuild time will increase when a drive fails--which also increases the chances of a second-drive failure during the rebuild. But the main advantage of ZFS is that it protects you against silent data corruption, since the filesystem itself is checksummed.

You can also run ZFS on other operating systems, but OpenSolaris is always the most up-do-date version since it takes a while to port new features to the other platforms. If setting up an OpenSolaris box seems a little more work than what you want, FreeNAS seems to be the next best thing, in terms of ZFS support.

On the Linux side, ZFS is not supported in the kernel (only as a user-level driver), but there is also a new filesystem under development, called btrfs. Unfortunately, there is no stable release of btrfs, as of March 2010.

Backup

For your offsite backups, it might be more cost-effective to pay for a service like CrashPlan, Carbonite, or Mozy. It's very, very easy to configure any of these to automatically backup your files. Of the three, CrashPlan has the best backup and recovery features (and even allows you to backup to other remote computers for free), while Mozy's recovery methods are either expensive or very inconvenient (if you want to download a Mozy backup, you have to wait for your job to be queued up and bundled into a zip file). I haven't personally had any experience with Carbonite.

Note that you shouldn't depend solely on an offsite backup--if you backup to the cloud or some other offsite computer, you should also have a local backup.

The Drobo reviews I've seen noted poor write performance, but if you're just using it as a nightly backup drive, it might be sufficient.

Backup rotation

If you want to rotate backups between a local and off-site location, you need at least 3 backups to guarantee one is always local and one is always safe at the off-site location. The third is either in-transit or at one of the other two locations at any given point in time.

ROBOCOPY vs. CrashPlan

ROBOCOPY will cause more wear and tear on your hardware, since it has to read every file during every backup. It's not clear to me whether it only copies changed files or if it copies all files. If ROBOCOPY fails for some reason, it may not be apparent that it has failed, unless you have set something up to reliably report its backup status.

CrashPlan monitors your hard drive for changed files, and only copies the changed files. Since it actively monitors changes to disk, it does not need to read every file in your backup source. CrashPlan automatically e-mails you to notify you how long it has been since the last backup, and how much data was transferred during the last backup.

That said, keep in mind that CrashPlan doesn't have to replace your ROBOCOPY backup scheme. You can use CrashPlan to supplement whichever other backup scheme you happen to choose.

Related Question