To address your question & a bit on enabling gpu decoding with mplayer or vlc.
Mplayer
The easiest way to see if you are getting gpu decoding thru vdpau is to do what you've done - run mplayer from a terminal. Atm what you've shown indicates you're not getting it.
The requisites for mplayer are:
mplayer or mplayer 2
nvidia-current or the nvidia drivers directly from nvidia, the nvidia-current package will do.
libvdpau
Make sure you've meet the above, then run the command you did before, though make sure the video is h.264
Ex. of terminal output, removed irrelevant lines
$ mplayer -vo vdpau -vc ffh264vdpau /home/doug/Videos/sintel.mp4
MPlayer2 UNKNOWN (C) 2000-2011 MPlayer Team
Playing /home/doug/Videos/sintel.mp4.
Detected file format: QuickTime/MPEG-4/Motion JPEG 2000 format (libavformat)
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (aac), -aid 0, -alang und
VIDEO: [H264] 1280x546 24bpp 24.000 fps 1597.2 kbps (195.0 kbyte/s)
==========================================================================
Forced video codec: ffh264vdpau
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU))
VO: [vdpau] 1280x546 => 1280x546 H.264 VDPAU acceleration
[vdpau] Got display refresh rate 59.991 Hz.
If you do support/get vdpau thru mplayer you can set a vc= line in ~/.mplayer/config to auto check & use the appropriate ffvdpau* decoder for supported codecs
Another way to check is to use vdpauinfo. The package is only available in 12.04 but can be installed in 11.10. To do so go here, download & install with gdebi, though software center may install for you if you wish to try
http://packages.ubuntu.com/precise/vdpauinfo
To use open a terminal & run vdpauinfo
Ex. of some of the output of vdpauinfo on a laptop with basic vdpau support
$ vdpauinfo
display: :0 screen: 0
API version: 1
Information string: NVIDIA VDPAU Driver Shared Library 280.13 Wed Jul 27 17:18:15 PDT 2011
Video surface:
name width height types
-------------------------------------------
420 4096 4096 NV12 YV12
422 4096 4096 UYVY YUYV
Decoder capabilities:
name level macbs width height
-------------------------------------------
MPEG1 0 8192 2048 2048
MPEG2_SIMPLE 3 8192 2048 2048
MPEG2_MAIN 3 8192 2048 2048
H264_MAIN 41 8192 2048 2048
H264_HIGH 41 8192 2048 2048
VC1_SIMPLE 1 8190 2048 2048
VC1_MAIN 2 8190 2048 2048
VC1_ADVANCED 4 8190 2048 2048
Vlc
If you have nvidia hardware & can't or don't have vdpau support then don't bother with vlc which uses vaapi & is tricker to enable. Additionally the performance of vaapi on nvidia is very much hardware dependent. I've enabled here but don't use, ffmpeg multi-thread is better with my hardware in vlc.
This is just the basics of getting vaapi & enabling in vlc. For specifics you'd need to search & or ask how .
The requisites for vlc are:
vlc & ffmpeg shared libs that have vaapi support, 11.10 is good there.
libva1, also install libva-x11-1 vainfo
vdpau-va-driver
Your biggest issue may be that ubuntu's versions of libva* & vdpau-va-driver aren't new enough, you'll need to get further advice there. In 12.04 they are adequate
Running vainfo in a terminal should tell you where you stand
Ex. here of vainfo
$ vainfo
libva: libva version 0.32.0
Xlib: extension "XFree86-DRI" missing on display ":0.0".
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/dri/nvidia_drv_video.so
libva: va_openDriver() returns 0
vainfo: VA API version: 0.32
vainfo: Driver version: Splitted-Desktop Systems VDPAU backend for VA-API - 0.7.3
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointVLD
VAProfileH264High : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
To enable gpu decoding in vlc thru vaapi & ffmpeg, two ways
From a terminal, no need to set in preferences
vlc --ffmpeg-hw /path/to/supported/video/name
To set in vlc's gui, the prior posts are incorrect as to how, actual way is shown in screen
Tools > Preferences > Input/Codecs > check the box, reading up on "Skip H.264 in-loop deblocking filter" may prove worthwhile
Ex. from terminal of vlc gpu decoding thru vaapi, using the repo vlc in 12.04 though 1.2/1.3/2.0 are preferred versions, a ppa for 2.0 should be available if desired
vlc --ffmpeg-hw '/home/doug/Videos/sintel.mp4'
VLC media player 1.1.13 The Luggage (revision exported)
clipped .....
libva: libva version 0.32.0
Xlib: extension "XFree86-DRI" missing on display ":1".
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/dri/nvidia_drv_video.so
libva: va_openDriver() returns 0
[0x863468c] avcodec decoder: Using VA API version 0.32 for hardware decoding.
Best Answer
Totem can no longer be used as a front-end for Xine, and going forward will probably be even more tightly tied to Gstreamer.
Xine is not as popular as it once was (in fact, it's pretty much dead), so finding a "modern" GUI for Xine is essentially impossible, unless someone writes a new one. I would suggest sticking with VLC, or Gnome-Mplayer.
Unfortunately your options, though varied, will not offer much of anything new and exciting, so to speak.