linux – Recommended Ways to Defend a Remote *nix Install from a Hamfisted Admin

Once in a while (often a long while), I have a difficulty where I execute a command that completely screws up a Linux machine.

Most recently, I accidentally mounted-again the root partition (thinking it was the new USB drive I had just formatted), and then proceeded to recursively chown the partition to myself (again, just trying to grant myself user-access to the USB drive). As soon as I realized what I had done (in mid-progress), I aborted it, but the damage was done. Many core programs were no longer root owned, so the machine was essentially in a zombified state. Some user functions (ssh, rsync) still functioned, but administration level stuff was totally locked-out. Couldn't mount, umount, reattach to screen sessions, reboot, etc.

If the machine were in the living room here with me, "repairing" it (reinstall) would have been trivially easy. But it isn't. It is in my brother's house. He's not big on me walking him through repairs/reinstall, and I understand that. So, I'm going over in a few days to fix the damage I did (and hopefully install something more admin-screwup-resistant).

I say all that to ask the question: What are the recommended ways of hardening an install against admin-hamfistedness?

Things not considered, or considered and dropped quickly:

1. Harden the administrator to not execute stupid commands: A great idea, but won't work, because as a human, I occasionally will do things that I realize after-the-fact are a bad idea. What I'm looking to do is out-think myself in advance, so when I do something stupid, the machine will refuse, and I'll realize "Oh crap! That could have been Very Bad (TM)! Let's not do that again."

Things I've considered:

1. Mount the root partition read-only: Would protect the root from changes, which might have negative affects if parts are expected to be writeable, and aren't. Also wouldn't necessarily protect the partition from being mounted again somewhere else as read-write.
2. Use a compressed-readonly root image of some sort with a union-like writeable layer above it so no changes are ever really made to root, and a reboot clears any screw-ups: This would be OK/good if no changes ever need to be made to root, and maybe /etc could be reloaded/populated from a persistent file somewhere else.
3. Use btrfs with regular (daily, maybe) snapshots, so that if an error is made, recovery is easier: Might still be sub-optimal as it would require direct user intervention, and I don't know that I could walk someone else through the changes to roll back the oops.
4. Use a more "live"/"embedded" Linux/BSD distro designed more with stability/predictability/security in mind instead of a more generic distro

As things stand now, I'm likely to use option 4 to install a somewhat more limited system than the full Debian install I had been using. But as just a file server and torrent client, it should work fine, and as a remote machine, defending the machine from myself is a pretty big asset.