No. X is the only usable GUI on Linux.
There have been competing projects in the past, but none that gained any traction. Writing something like X is hard, and it takes a lot of extra work to get something usable in practice: you need hardware drivers, and you need applications. Since existing applications speak X11, you need either a translation layer (so… have you written something new, or just a new X server?) or to write new applications from scratch.
There is one ongoing project that aims to supplant X: Mir. It's backed by Canonical, who want to standardize on it for Ubuntu — but it hasn't gained a lot of traction outside Ubuntu, so it may not succeed more than Wayland (which was designed for 3D performance, not for security) did. Mir does aim to improve on the X security model by allowing applications limited privileges (e.g. applications have to have some kind of privilege to mess with other applications' input and output); whether that scales when people want to take screenshots and define input methods remains to be seen.
You can run a few graphical applications on Linux without X with SVGAlib. However that doesn't bring any extra security either (in addition to numerous other problems, such as poor hardware support, poor usability, and small number of applications). SVGAlib has had known security holes, and it doesn't get much attention, so it probably has many more. X implementations get a lot more attention, so you can at least mostly expect that the implementation matches the security model.
X has a very easily understood security model: any application that's connected to the X server can do anything. (That's a safe approximation, but a fairly realistic one.) You can build a more secure system on top of this, simply by isolating untrusted applications: put them in their own virtual environment, displaying on their own X server, and show that X server's display in a window. You'll lose functionality from these applications, for example you have to run things like window managers and clipboard managers in the host environment. There's at least one usable project based on this approach: Qubes.
Best Answer
Not very sure, but the closest I can think of is cgroups:
For more information, see one of: