QuickTime alone won't capture computer audio along with a screen capture, but combined with a free utility Soundflower, you can do this.
MacWorld has a step-by-step guide, but the basic idea is to install Soundflower and set your computer audio out to Soundflower, so system sounds play to it and not to the speakers. Then in QuickTime, when you set up a new screen capture, select Soundflower as the audio source. Then you can do a screen recording and capture system audio at the same time.
Note that this is the free but messy version. Software like Screenflow does this sort of thing very well, but costs $99.
Not exactly a solution, but some observations that might provide a clue:
- Issue also seen on my Mac mini (Mid 2011 - 2.7GHz Intel Core i7 + 8 GB DDR3) after upgrading from Lion to Mountain Lion
- QuickTime no longer saves the recording to ~/Movies/Screen Recording.mov, but instead saves it to some temp location and when done recording, you need to save/export it.
- Downgrading quality to medium quality didn't help
[Update 11/17]
Even my new Macbook Pro Retina 15" has this issue.
I happened upon a tutorial that showed someone using VLC to screen capture on a Mac and decided to give it a try. It turns out that functionality was removed from VLC on Lion and Mountain Lion and the reason was provided in this this thread: Mac OS Lion, VLC screen capture does not work:
Apple completely removed the API to capture the screen and replaced it with something totally different. Until anyone steps up and implements the new one, this feature is not available to Lion users. It remains accessible to Snow Leopard and Leopard users of course.
I'm not sure if this is also what's causing QuickTime on Lion+ to be super slow.
I did find an alternative which is working out really well for me: Voila
Available on the App store for $10 and records w/o performance issues. I was skeptical at first, so I downloaded its trial, and was extremely happy with the performance. $10 is definitely more than free, but depending on how often you need to record your screen, this is well worth the money in my opinion.
It supports both JPEG and H.264 encoding, but only records up to 25fps. For audio, it allows you to select which input and even allows you to use the computer audio (something QuickTime didn't allow and you had to use a separate utility like Soundflower).
Best Answer
Finally figured it out - apparently the Movies folder in my home directory had a weird permission set on it and things couldn't be written to it, which caused QuickTime to give a very descriptive message "The operation could not be completed" and saving to a different folder (or after fixing the permissions on Movies), worked.