I would like to suggest something radical. I got this idea from StarTrek : Deep Space 9 (Call to Arms)
A minefield was set up around a wormhole to prevent the Dominion
Forces and the Jemhadar from passing through. Each explosive mine was
armed with a replicator that allowed a mine to recreate another mine
after it explodes. Hence, the minefield stays up indefinitely.
With that DS9 analogy I bring your an interesting idea.
You set up 2 initial read slaves with MySQL Slave with innodb disabled
[mysqld]
skip-innodb
This is optional. My preference is an all-MyISAM slave to do reads because it is faster for reads than InnoDB for small datasets. Should you choose to go with InnoDB, make sure you relax the ACID compliance with this
[mysqld]
innodb_flush_log_at_trx_commit = 0
In the event of a crash, just destroy the Slave and spin up a new one
We'll call the Slaves S0 and S1
Here is something else: have this in /etc/my.cnf in S0
[mysqld]
innodb_max_dirty_pages_pct = 0;
innodb_fast_shutdown = 0
These will help S0 shutdown fast with completely flushed data.
The following is what you must script in the replicator process
When you need to generate a new slave (we will call it S2), here is what you must do
STEP 01) On S0, run service mysql stop
STEP 02) Install the same version of MySQL that S0 has into S2
STEP 03) On S0, scp /etc/my.cnf S2:/etc/.
STEP 04) On S2, you need to change the server-id of /etc/my.cnf in S2 to a Unique Value (suggestion : use the 2nd,3rd, and 4th octet [without the dots] of the private IP of S2)
STEP 05) On S2, either remove or comment out the innodb_max_dirty_pages_pct = 0 from /etc/my.cnf
STEP 06) On S0, run rsync -av /var/lib/mysql S2:/var/lib/mysql (Note: If you have to spin up 5 Slaves, run the 5 rsyncs at this point)
STEP 07) On S2, service mysql start (MySQL Replication start immediately where S0 had left off)
STEP 08) On S0, run service mysql start
Once you create the replicator script, you can use it spin up MySQL on new Slaves.
Meanwhile, S1 is available for SELECT queries and continues replicating.
S0 is used to recreate a new Slave.
If you are doing this in conjunction with spinning up Amazon EC2 or some other Cloud DB Servers, check with your System Administrators on any Linux commands/API that allows you to spin up a DB Server. Then, you apply the replicator to the newly generated DB Server. Even better, you can incorporate the Db Server creation API in your Replicator Script.
CAVEAT
If you have to spin up dozens or hundreds of Slaves, all you need to do is have 10 Replicator Slave Servers (S0 - S9) and have 10 copies of the replicator script operator on different Replicator Slaves.
Best Answer
The nagios monitoring scripts for mysql should be a good start.
If you want to monitor your database with this kind of email alerts, you should really try to setup nagios or some similar software as this is exactly the purpose of a monitoring software.