On second thoughts I'm thinking it'd be a bit redundant anyway as the database connection credentials will need to be stored on disk for the rails app anyway!
You've essentially summarised your problem.
Even if you could encrypt the MySQL store - "all" they have to do is compromise your web server, and they get your DB credentials and can happily dump your database. The fact really is that if they can get the same level of access as you at any point - all your security efforts have gone down the drain.
Here are some general tips, but they don't replace your professional security advice:
- Minimise access. Never have a user that can access the database server from any IP address, you're just begging for a brute-force attack. Instead, tweak the external firewall (e.g. the one EC2 called "Security groups") so only your web server can even connect to your DB server. The less people that know your VPS is a DB server, the better.
- Make sure everything else is secure. That means your email passwords, security answers, and so on. If they can get access to your EC2 account or your email account, you've just completely wasted your time securing the instance. Social engineering attack are all the rage these days, don't let yourself be next.
- Ensure your VPS instance is bare-metal. Don't have software on there that doesn't need to be, that even goes for sendmail (this is just an example of how, back then, you might have though "sendmail can't hurt to leave on!"). Anything that is there, keep updated frequently and watch out for security notices.
- Try to avoid having any services in general that are public facing, especially something like SSH. If you must, disable password authentication and put SSH on a non-standard port (then connect to it like
ssh -p <port> user@host as normal.
Having said all of this, it won't make your case to the client any better. You'll still say "it's as secure as it can be" but not "the data is encrypted on disk".
Your problem boils down to access control.
The first defense I'd propose is to simply deny access to the untrusted users. If they can't get into the database, they can't query the database and get at the sensitive data.
If they must be allowed to access the database server, you can look at either explicitly granting them read permission to the tables they must access to perform their job tasks. Alternatively, you can leave their current permissions alone and simply revoke their ability to query tables or columns within those tables with sensitive data.
The reasons I'd start with this approach is that there is overhead to encryption. There is key management as well as the CPU cost of encrypting/decrypting the data. As I understand it, there's also going to be a performance cost of looking up data as the encrypted data if it's in an index, the encrypted value will be indexed and not the source value. The net result is that 2013-06-11 and 2013-06-12 could have been stored in contiguous disk locations but once they're encrypted they could be on opposite ends of the disk and your simple range query which used to perform well now sucks hind teat.
All of that said, Steve Jones had a good presentation on the Encryption Primer. I used some of that content plus this article on encryption in the DB but not in the cube to get our encryption stuff up and running.
As others have pointed out in the comments and other answers, despite having Encryption as part of its name, TDE, isn't going to protect the data in the database itself. The purpose of TDE is to protect your backups from unauthorized access. It has nothing to do with encrypted packets flowing across the network or automagically creating PCI compliance. It simply ensures that if our offsite backup tapes are lost, stolen or simply snooped, people will not be able to use them without the key.
The other point to access control that I failed to mention is physical access. If the concern is the junior admin accessing data they shouldn't, then in addition to denying them access to the database, don't overlook basics of preventing them from accessing the machine itself---either through remote desktop or physically logging onto the machine. Once they are able to get onto the box, there's nothing preventing them from restarting SQL Server into single user mode and removing your access controls.
Best Answer
Adding such an additional secret - and, ideally, storing it securely in an HSM - is indeed a normal practice. It's not common, exactly; many platforms are using hashes alone (and often poorer hashes than bcrypt). But an additional layer definitely makes things harder for the attacker - if done right.
Some further advice: roll as little of the implementation yourself as possible. Instead, find well-established libraries to do as much of the crypto as you can (because it's hard to get right - and getting it wrong can be very bad indeed).
Note that bcrypt has some limitations - notably that the maximum plaintext length allows is 72 bytes. It also is not memory-hard, and therefore is subject to attack with (relatively) lower-cost specialized hardware with many cores in parallel. The modern options - scrypt and the Argon2 family - are resistant to low-memory parallelization. Look for well-baked implementations of those. Also, use bcrypt with as high of a work-factor cost as your users (and projected hardware) can tolerate (cost 12 is often where it starts to get close to the break-even point, but YMMV; test on your hardware).