Somewhere around late 2014 is got very reliable for me. I don't know if iOS changed, OS X changed or both, but before then I continued to have the problems you describe with Mountain Lion 10.8.4 and earlier.
Of late, this has been quite solid and at worst, toggling the connection in network preferences on OS X fixes any connection issues with iOS hotspot being solid in general.
Before the fixes, the problem seems to be a timing issue.
What I've found is that if the connection light doesn't show green, I want to disconnect the device and turn off tethering on iOS. After waiting 5 seconds or so, I turn on tethering again on iOS and be sure to leave the tethering screen showing. (don't leave the settings app) I then reconnect the device and listen for it to signal that it's connected with the chirp iOS makes when connecting to a computer. If the light doesn't go green in system preferences, I toggle the Personal Hotspot off, count a few seconds and turn it on.
Occasionally, something just isn't correct on the Mac end and I switch to Wi-Fi tethering and delete the interface in System Preferences and then start over with the set up.
I wish I could say this worked reliably for me, but it seems to be fidgety and prone to errors and lock up.
Best Answer
What if you just switch off Bluetooth? You're Mac will disconnect (obviously) and then you'll have removed your iPhone from your Bluetooth preferences.
If that doesn't work, you can do it the hard way and delete following files from ~\Library\Preferences:
Please be advised you'll delete all your stored Bluetooth devices !!