I used to see this occasionally when I worked at the Apple Store. If it gets really hot while it's charging but there's nothing on the screen, it's most likely still alive. It could be hard locked up by a runaway process from the app you were using that can't quit .
For phones with a removable battery, you might just pull the battery for a few seconds and replace it and then restart the phone. You can't pull the battery on the iPhone, but sometimes attempting to restart it while it's on power will break it out of the black screen. Try plugging it into your computer and holding down both the sleep/wake button and home button (same as how you tried to reset it before) for at least 15-30 seconds and see if you get the Apple logo.
The other thing you might want to try is putting it in DFU mode and then rebooting it normally again. You can find the steps here. Don't try anything crazy, just boot it into DFU (screen should be black) and then restart it how the instructions say.
If it's still not restarting, take it off the charger and let it sit overnight to try and allow whatever charge is left on the battery to run down. Then, plug it in to a power source and attempt to restart it by holding the sleep/wake and home button. This is really tedious to wait on, but I had people bring me phones that could magically be rebooted after sitting all night until the battery ran out.
One last method, as a last resort: If you have a backup, you might want to attempt to put it in recovery mode and restore it. Sometimes it will go into recovery even if there's nothing happening on the screen. To do this, plug your cable into the computer, and then hold down the home button on the phone. Keep holding home and plug the other side of the cable into the phone. Continue holding down the home button until you see the message in iTunes that the phone is in recovery mode. RESTORING YOUR PHONE WILL ERASE ALL YOUR DATA so don't do this right off the bat if you don't have your stuff backed up. Try less the less destructive methods of troubleshooting first. You can also do a restore from DFU mode if iTunes will recognize the phone.
I'm pretty sure your phone isn't going to explode, but if you just can't get it going again, it might be a good idea to take it to an Apple store if you can get to one. There's never a charge to look at your phone and you're not under any obligation to have service done on the phone if there will be a fee. But if you want to, you can check the status of your warranty (a very few of the 3GS phones are under warranty still, especially if there's been AppleCare on the phone) at selfsolve.apple.com.
If the phone can be powered on eventually, they have a diagnostic tool called Behavior Scan that they can run at no charge that will tell you if your phone is overheating or what particular apps are hanging the phone up. However, keep in mind that if you've recently restored your phone it won't return much useful information.
It could be a poorly written app that's hanging your phone, or it might need a software update or a restore. If you can get it going again, it would be a good idea to check if you're running the latest software and that your apps don't need updates either. These things tend to be more software related than hardware, but if it keeps shutting off and a factory restore won't fix it, you might be looking at some bad memory in the phone or a logic board issue so keep that in mind.
I am assuming that the phone has not been wet or been subjected to physical damage. If it has been, that might be a factor as well. I'm also assuming the phone is not jailbroken. If it is, be careful with restoring it because you can end up bricking your phone if you're not paying attention.
Best Answer
Sometimes when you run some games that is not compatible with your iPhone, it will use more power than it can to the CPU and GPU and then it will crash. Try to verify if the game is compatible with iPhone 3GS