Sorry, you can't SLI/Crossfire link an Nvidia card with an ATI one.
As for having both cards in the machine, and simply plugging another monitor in... This should work fine. Windows 7 improves on Windows Vista's graphics architecture and allows graphics cards from different vendors.
The first place to check is Nvidia's System Requirements for 3D Vision. Without using SLI (which would require two or more of the same cards), you can't do what you're asking. This is just due to the way it works - rendering such a large image, and then sending it to another discrete video card before displaying it would be such a performance hit it wouldn't be worth it.
While this uses the 3D Vision technology, you don't have to run games in 3D mode - you can run them in 2D mode as well. In this mode, you also don't have three desktops, but rather one triple-wide desktop (at the desktop level). See this thread for more details.
Do note that the game must also support the 3D Vision Surround Technology. Note that using three monitors on the desktop level is a completely different story (if you want discrete desktops as opposed to a triple-wide one), and you can see details for this below.
Additional information: As you already know, you need to use an additional video card to get the triple monitor setup working. Additionally, if you wanted to use SLI/Crossfire in this setup, you would have to check Nvidia's website to see if triple monitors are supported (e.g. I had 2 x 8800 GT's in SLI, and had to disable SLI to get triple monitors to work - this does work with some newer Nvidia cards however). See this page for more details.
Using two video cards in non-SLI is a completely different story, and quite possible with any combination of cards. My recommendation is to use your one center monitor as your "main" monitor (and have the 560 be that monitor's hardware acceleration), and plug the other two monitors into the 260. That way, 3D applications that output to your main display won't take a performance hit (those running on the additional monitors will - you might want to put one of them back on the main 560 if you need something else hardware accelerated).
The core requirement from Nvidia's website to use an additional video card to drive an additional monitor is:
Additional monitors (up to 6 monitors total enabled) may be enabled by using either a motherboard GPU and/or a PhysX capable graphics card (GeForce 8 series or higher with at least 256MB of memory) that does not have the same GPU as those that are SLI enabled. More information regarding multi-monitor in SLI can be found here.
Long story short: With the hardware you have, it's impossible. If you got another GTX 560, then it would be possible.
Best Answer
Of course only one card was maxing out the single GPU, you can't run SLI with different GPU cores.
Save up and get another GTX 560 Ti if you want to increase triple monitor 3D performance too. The only ability you get by adding a lower performance graphics card is the ability to use three monitors, that's it.
You can also use a slower card to perform the PhysX rendering. Also, if you use any CUDA-specific applications, you can offset them to one particular GPU if you needed to.
If your second GPU happens to be SLI-compatible with the first one (as with Crossfire, which only happens when they share the same GPU core), when SLI/Crossfire is enabled, your performance will be limited by the slower card (or rather, the card with a lower amount of memory or lower clock frequency).