Removing voices
All voices are in /System/Library/Speech/Voices. Just like most other Library items in OS X, they are inherited, which means you could install them in /Library/Speech/Voices and ~/Library/Speech/Voices.
Removing items from /System/Library is indeed usually not a good thing to do, but here, it does not have any practical consequences beyond making the deleted voices unavailable. Indeed, the list in the System Preferences is dynamically generated from the items found in the said folder, just like the Sound alerts, or the wallpapers. You can confirm this by adding / removing / renaming elements in the /System/Library/Sounds folder, for example.
There's no other way to manage the speech items, but since this one is without consequences as long as you leave at least one voice, it is actually pretty nice to simply manipulate the filesystem. I have personally long removed Alex, along with other /System/Library and /Library items.
Installing voices / what to back up
As you underlined, Lion provides a “Customize” item in the list of voices that allows you to download voices in many languages. Therefore, if you ever delete the original “Alex” voice (the heavy one, 400+MB), you can download it again through this menu.
The “funny” voices, however, are not downloadable. You'll need to back them up if you want to remove them but keep a way to get them back (or you could use a package manager like Pacifist to extract only them from the Lion installer package, but that's a lot of hassle for not much). Nevertheless, considering they weigh, all together, less than 35MB, I'd let them in place.
Please note that you must leave at least one voice for the “Customize” item to be accessible. So, even if you want to remove all voices, leave at least one there, or you won't be able to install any again. “Fred” is the lightest “serious” voice, so I'd advise leaving it in any case.
Size note / a bit of history
As for the relative sizes, Alex is very heavy because it is of much higher quality, and is actually from a totally different generation than the others: it shipped with Leopard, while others have been around since Mac OS 7 (had a lot of fun with them, actually ;) ). Of course, Alex would have filled a full hard drive at the time. There's no "logic" that's shared between voices in those packages, hence the heavy weight of installing new “modern” voices in other languages.
Best Answer
Source: https://discussions.apple.com/thread/2763273