Not exactly a solution, but some observations that might provide a clue:
- Issue also seen on my Mac mini (Mid 2011 - 2.7GHz Intel Core i7 + 8 GB DDR3) after upgrading from Lion to Mountain Lion
- QuickTime no longer saves the recording to ~/Movies/Screen Recording.mov, but instead saves it to some temp location and when done recording, you need to save/export it.
- Downgrading quality to medium quality didn't help
[Update 11/17]
Even my new Macbook Pro Retina 15" has this issue.
I happened upon a tutorial that showed someone using VLC to screen capture on a Mac and decided to give it a try. It turns out that functionality was removed from VLC on Lion and Mountain Lion and the reason was provided in this this thread: Mac OS Lion, VLC screen capture does not work:
Apple completely removed the API to capture the screen and replaced it with something totally different. Until anyone steps up and implements the new one, this feature is not available to Lion users. It remains accessible to Snow Leopard and Leopard users of course.
I'm not sure if this is also what's causing QuickTime on Lion+ to be super slow.
I did find an alternative which is working out really well for me: Voila
Available on the App store for $10 and records w/o performance issues. I was skeptical at first, so I downloaded its trial, and was extremely happy with the performance. $10 is definitely more than free, but depending on how often you need to record your screen, this is well worth the money in my opinion.
It supports both JPEG and H.264 encoding, but only records up to 25fps. For audio, it allows you to select which input and even allows you to use the computer audio (something QuickTime didn't allow and you had to use a separate utility like Soundflower).
To get the display back without rebooting, I've had success doing this:
- Disconnect the display
- Wait until the display goes to sleep (set the timeout to 1 minute first)
- Reconnect the display
To avoid the issue altogether, I've gotten into the habit of disconnecting the display before I leave the Mac.
Best Answer
The (Ugly) solution that works is installing gfxCardStatus, Choosing "Integrated Only" and screencasting my laptop monitor without any external monitor.
It's is ugly indeed - will welcome any proper solution.