First of all, a Display Manager (or DM, e.g. xdm, gdm, kdm) is not the same as a Desktop Environment (or DE, e.g. GNOME, KDE, XFCE).
The Display Manager takes care of graphical login, and decides (or lets you choose) what session to run. Or what session*s* in case you choose the "switch user" menu option.
A Desktop Environment is basically a collection of programs (display manager, window manager, session manager, panels, configuration tools, etc.) and libraries (e.g. Gtk) that intend to give a consistent and integrated environment to work in.
A Window Manager manages windows: where to place them, move them, resize them, minimize/maximize them, tile them, etc.). It also handles the shortcuts to do those things. In some cases the window manager also paints the borders of windows, in other cases this task is off-handed to a "window decorator".
The "Run Application" dialog in GNOME is part of gnome-panel
, but in another DE it could be another part of the environment.
Who is in charge of painting windows etc. depends; in case there is a "compositor" in use (often part of the window manager, e.g. in Compiz) then the compositor paints the windows on the screen, otherwise (as was usual in the past) it's the X-server doing that.
The Main Menu(s) are put on the screen by a part of gnome-panel, but the data used comes from a bunch of files in /usr/share/applications/
(possibly combined with an equivalent directory in your home for personal changes). Those files have a structure defined by FreeDesktop.org (a platform for different DEs to collaborate on common infrastructure), so that GNOME and KDE know about the same programs (but still can show them differently, and in some cases prioritize "native" programs over "foreign" ones).
And yes, different users can use a different session configuration (and can even define their own ones). In GDM, try the Session drop-down for the available choices.
Furthermore, it is possible to mix & match several components, but that will sometimes result in less cooperation and a loss of "smoothness" in how things work. One very well known example where things get exchanges is of course Compiz, which replaces Metacity if you want fancy desktop effects. But there are lots of other changes possible.
I asked a similar question once. It is definitely a good idea if you find yourself ignoring the miscellaneous tools and features that come with the desktop environment.
The solution is, you don't need to install a desktop environment (or anything you don't use), just a window manager of your choice. I (also) asked another question about lightweight window managers. The summary is, if you want lots of effects go Compiz, otherwise choose a lightweight one like dwm.
Regarding Nautilus, do you really need it or some of it features? If not then you should probably switch to something else, maybe Thunar.
Best Answer
I was looking for an alternative to Gnome which would be built on Mutter, that's how I got here. If you find such thing please let me know.
Here is the documentation for Mutter, it contains a reference for developers, it might help:
https://developer.gnome.org/meta/