Certain upgrades are possible, but it’s not that simple.
Accelerate your Mac is a nice source of information, you can find them here.
ANandtech did change the CPU of a Mac Pro.
OWC Mac Sales does have memory upgrades (there are others).
Video card: Unless you want to experiment with modified firmwares, your only choice in that model is the ATI 4870, which is considerably faster than the regular NVidia (it’s very noticeable in x-plane and 3d applications) but still lagging behind the “latest offerings”.
According to Wikipedia: "The Mac Pro, as with other Macintosh platforms, requires Mac OS X firmware. That is to say, a PCIe video card designed exclusively for other operating systems will not work properly under Mac OS X without appropriate drivers and/or firmware”
Sadly, no other video card has that, other than the Apple’s offerings. But If you google around there are some posts about people who managed to make other cards work.
Forget about upgrading your Motherboard, as it’s a custom design.
Hard drives/SuperDrives are generic and can be replaced, as you may have already guessed.
Any other component is probably “custom” designed for a Mac Pro (the power supply for example) and you won’t be able to get away with it…
Is it worth?
I have an early 2008 Mac Pro with 2x4 Intel XEON 2.8 and 10GB RAM. I’ve upgraded the RAM (from the lame 2GB to 10 @ OWC liked above), and I ordered the machine with the NVIDIA card (8800? I don’t remember), which was the only option other than the stock ATI2600 or something lame like that.
A couple of years later, the NVIDIA died (the Apple guy told me it had bugs and used to fail), so I got the new ATI 4780 512MB (it was 100€ less expensive than the out of warranty NVIDIA apple replacement). The difference was really noticeable in 3d apps.
I haven’t thought about upgrading the CPUs because getting faster XEONs (I wouldn’t be able to upgrade to Nehalem or anything new) won’t really make a big difference for me.
I use a SSD and 4 Hard Drives in the bays, if that counts as an upgrade (my original drive was 750gb, up from the stock 500).
I brought it with the Bluetooth/Wifi module (it was optional back then) and tho I haven’t used it a lot, it came in handy in a few occasions.
OS X Doesn’t support SLI/Crossfire for using more than 1 video card in parallel (you can install more than one, but it’s not the same thing), so forget about getting two super video cards for running crysis y super-top-ultra-maximum quality.
Conclusion
Yes, if you upgrade the RAM to something other than the 2GB and yes to the video card if you do 3d intensive stuff. Exposé will work fine on the NVIDIA8800 you have, but X-Plane (among other 3D things) will be noticeably faster with a new video card.
You're better off using an Mac-supported cards from Apple or others.
Apple's card are seriously outdated, both 5770 and 5870 are slow card in today's standard.
There "should" be a renew of Mac Pro some day, you can wait for that.
There are a cards like the Quadro K5000 that is Mac-compatible. But NOT out of the box. You need to install the driver first, and then install the card.
Also, support for AMD HD 7 series cards ARE NOT IN YET in 10.8.2. I had one of those in my old Hackintosh running 10.8.3, and it's still recognized as "AMD Radeon HD 7xxx series", so I will say wait for it.
Don't make any moves before AMD 7 series is out, maybe Apple or third-party will release cards for it.
Good luck!
Best Answer
At best? 100% nothing. At worst? Crash your whole system and make it unusable or even unbootable.
While Mac OS X and Linux are both different “flavors” of Unix, you can’t just grab parts from one OS and just install it into the other OS and expect them to work.
That said, if there is source code for the driver somewhere, you could potentially compile them for Mac OS X using the Xcode suite of development tools.
But that is one big if. The reality is compiling Linux application source code on a Mac OS X system from a Linux system is a crapshoot. Sometimes it works and you get to use nice Linux tools on a Mac OS X system. But sometimes it just won’t work. And that is at the application level. On the driver level it’s even less plausible or doable; and even if it was doable it would be difficult to do that at best.
That said, there seems to have been an open source effort to compile Linux Intel drivers for Mac OS X as seen here. But note that the versions of Mac OS X referred to are Mac OS X 10.4 and 10.5; nothing more modern.