I've found that "New Screen Recording" functionality in QuickTime Player combined with iMovie editing capabilities is enough for my needs. Maybe you can give it a try? Best of all, it's already installed on your Mac. Unfortunately, I can't compare it with Camtasia, as I haven't used it.
Unfortunately, there is not a cheap good or ideal way to do this.
The graphics card is a key integrated part of the logic board on the MacBook Pro, there really is no separation of it from it to do an upgrade, short of replacing the entire motherboard, and even then you would be limited to what was available at the time for that generation of MacBook Pro.
See that orange highlighted square, thats the NVidia GeForce pictured below:
Honestly, the Matrox route is probably the most affordable workable solution to this problem.
This MacBook Pro also lacks an ExpressCard/34 slot or Thunderbolt so you really can't go the route of an external graphics card. And even if you could you would be talking over $500 USD just to get the external PCI express chassis, for example Magma's ExpressBox 1 starts at $829 for just one external PCI Express slot, you would still need to spend another ~ $200 to buy a mac compatible graphics card.
Your last option is to maybe reevaluate if this MacBook Pro is the right machine for your needs. Would a newer model MacBook Pro with Thunderbolt solve this problem, maybe maybe not. If you don't have two Thunderbolt Displays, you are kind of still out of luck hooking up dual displays to even a Thunderbolt MacBook Pro, with out the need for expensive external adapters.
Would an iMac, Mac Mini, or Mac Pro better suit your computing needs, complimented with a basic MacBook Air or iPad of some kind for your mobility needs.
Best Answer
The reason for the “stuttering” could be a combination of different factors.
First is of course your computer’s power and video card. The more power you have, the more the number of FPSs your computer can “record”.
Despite that, QuickTime is not known for its “fast” recording, so I suggest you give a try to different alternatives. This comparison is a little bit “outdated” because some (if not most) of the applications mentioned have been updated and have fixed or changed some of the problems they had, however, it should be a starting point for your final QUickTime Replacement. This other link has a very similar list, but rather than a review is a list with a short description of the products.
For reference, the most important Screen Capture tools for OS X (or to put it in another way, the “most used ones”) are (in no particular order): iShowU, Snapz Pro X, ScreenFlow and one that is missing from those lists (because it’s a new player in the Mac world): Camtasia for Mac (which I’ve been betatesting and it’s quite good and in par with the ones I’ve mentioned). Some features missing from Camtasia 1.0 have been addressed in an update and I know they are working on a newer version after getting tons of feedback in the Beta forums (which I can’t disclose because I participated).
I’ve personally tried those four I’ve mentioned (i have a Snapz Pro license) and I think they more or less are the same, but some were (when I tried them) faster than others but lacked certain features. All in all, the “best” doesn’t exist, they all have strengths and weaknesses so your best bet is to give them a try and see if one does the job.
Last but not least, remember that quality also plays an important factor in CPU overhead when recording, try to keep your CPU load low and reduce the recording quality as much as you can afford.