Quicktime record screen without the overlay

Not exactly a solution, but some observations that might provide a clue:

• Issue also seen on my Mac mini (Mid 2011 - 2.7GHz Intel Core i7 + 8 GB DDR3) after upgrading from Lion to Mountain Lion
• QuickTime no longer saves the recording to ~/Movies/Screen Recording.mov, but instead saves it to some temp location and when done recording, you need to save/export it.
• Downgrading quality to medium quality didn't help

[Update 11/17]

Even my new Macbook Pro Retina 15" has this issue.

I happened upon a tutorial that showed someone using VLC to screen capture on a Mac and decided to give it a try. It turns out that functionality was removed from VLC on Lion and Mountain Lion and the reason was provided in this this thread: Mac OS Lion, VLC screen capture does not work:

Apple completely removed the API to capture the screen and replaced it with something totally different. Until anyone steps up and implements the new one, this feature is not available to Lion users. It remains accessible to Snow Leopard and Leopard users of course.

I'm not sure if this is also what's causing QuickTime on Lion+ to be super slow.

I did find an alternative which is working out really well for me: Voila

Available on the App store for $10 and records w/o performance issues. I was skeptical at first, so I downloaded its trial, and was extremely happy with the performance. $10 is definitely more than free, but depending on how often you need to record your screen, this is well worth the money in my opinion.

It supports both JPEG and H.264 encoding, but only records up to 25fps. For audio, it allows you to select which input and even allows you to use the computer audio (something QuickTime didn't allow and you had to use a separate utility like Soundflower).

Try repairing the disk permissions. Go to Disk Utility, select your HD and then click Repair and select Repair Disk Permissions. Hopefully, this will solve your problem. Allot around twenty minutes for the process.