The difference comes from the kernel and software installed.
Minimal: just the enough software to get you to a working command line. The kernel is normal.
Server: In pre-precise versions (i.e., before 12.04 LTS) you had the choice to install the server kernel, which works with lower latency and with different task scheduler. This install still gets you to the command line
Desktop: Normal kernel, default selection of software, and you get the graphical user interface.
They only differ from the installed packages, which you can later easily modify. If you want to install a server, choose the server version, if you want to use it as a desktop, choose the desktop version. If you need more performance, compile your own kernel from source.
Btw: if you want a lowlatency kernel, you can install the package for it (this has been available starting in precise). Your system will be more responsive but the CPU usage will be higher.