Switch away from full-screen DVD Player on a second monitor without having it exit full screen mode

When formulating a question, it's best to ask it accurately. In this case, the issue has nothing to do with Blu-Ray or DVD files. You aren't playing back from disc media. Instead, you are playing back from very high-bitrate files ripped or copied from the disc media. You are using the free VLC player to view the ripped files.

Let's analyze: a 50 GB file of Fast Five (130 min running time) would appear to be a copied, uncompressed file, since you can find .mkv or .mp4 files of the movie on torrent sites that run between 6 and 10 GB. So the file you're talking about is a very high-bitrate video file. You don't say, but I'm guessing it's a .m2ts file.

Two things:

1. The problem could very well be in VLC. Its support for .m2ts files is not optimized (due to licensing issues and other problems, the Blu-Ray format is not publicly well-documented).

2. I don't think there is anything wrong with your Mac. You are simply throwing a tremendous amount of data at it, and the software you are using for the video playback can't keep up.

Suggestion: obtain a 1080p version of the movie in .mkv or .mp4 format, then attempt to play that back with VLC. If the movie plays well, with acceptable quality and without lag, then you have your answer, and can watch future movies accordingly.

It sounds like OS X isn't properly detecting the resolutions your projector can support.

The best workaround would probably be SwitchResX. It's a shareware app (from what I can tell, no functionality restrictions) that allows you to set custom resolutions (and other display options). Set up one that you've confirmed your projector can support and give it a try.