Ny software for mac to create this particular 3d histogram

I assume that you mean 5000 observations. R will work, and RStudio will get you to use R with a bit more GUI than the native application. Commercial solutions like Stata -- which has a soft learning curve, in comparison especially to R -- will also work that kind of sample without any difficulty (by statistical standards, 5000 observations is a relatively small dataset).

The learning curve will matter if you want to reach beyond a mere plot; otherwise, a quick look at the documentation for line graphs should suffice (for R, check there and there, for instance). HTH!

I don't do a lot of LPs, but the last time I did I used QuickTime Player.app (Quicktime Player X); you'll find it in /Applications. To record your screen, select File > New Screen Recording.

The results were pretty good; recording Minecraft 1.7.3 gameplay on an about a year and a half old MBP, but I have a lot of RAM, so that certainly helps.

Edit:

It's worth noting that QuickTime Player X does not support loopback sound; meaning that the only sound recorded will be from the microphone, or virtual sound device providing loopback sound.

This may or may not be a good thing, depending on what you are doing.