apt-get purge only removes system-level configuration files (e.g. those created by debconf). It doesn't touch anything in your home folder -- you have to remove those manually. That makes sense if you think about it: there might be multiple users, who probably wouldn't want administrators deleting data out of their home folders!
Applications usually store configuration data in one of the following hidden folders, which you can delete if you're sure you have no use for the data:
~/.application~/.config/application~/.cache/application~/.local/share/application
Replacing application with the name of the app or package. Most applications don't store user files in those folders, and if you accidentally delete something out of your home folder that the system needs, it usually just gets re-created. Of course, it doesn't hurt to look inside before you delete them, just to make sure there's nothing you want in there.
For gnome 2 apps you might also want to run:
gconftool-2 --recursive-unset /apps/application
(again replacing application with the app name) which will get rid of your application preferences.
Update 12-14-2012
For gnome 3 apps, the gconftool-2 command above is replaced by:
gsettings reset-recursively [schema]
You can find the schema for the application with
gsettings list-schemas | grep application
Kinda embarrassing answering to my own question but found a solution how to remove orphaned packages that satisfies my needs.
First I made a new custom filter in Synaptic package Manager.
Settings -> Filters
Than Deselected All and checked only Orphaned check box and saved.
Now synaptic is listing all orphaned files. Note that synaptic will list files with no dependencies as orphaned too so don't remove anything you're unsure what it is (and if it is really orphaned).
Than I installed GtkOrphan using CLI sudo apt-get install gtkorphan which is a GUI for deborphan but adds option to remove packages too.
Once started GtkOrphan and has to be run as root, under Orphaned packages tab expand the Options and check the "Show all orphan packages, not only those in the libs section". After that GtkOrphan will report as "orphaned" all packages that have no dependencies too, for example libreoffice, ubuntu-desktop, ubuntu-restricted-extras, myunity and etc. Now you can check the list and using right-click context menu chose to hibernate each package you don't want to be reported as orphaned. List of hibernated packages can be accessed from right click context menu or View menu.
Always under Orphaned packages tab and expanded Options check the "Show uninstalled packages with orphaned configuration files". Before removing anything make sure you know what are the files you want to delete. In my case GtkOrphan found some old nautilus configuration files not shown in Synaptic Package Manager.
In case you delete some of configuration files you need re-installing the package should solve the problem.
Best Answer
To add a little bit to what Rinzwind and nickguletskii said,
completely removewill remove configuration files in/etcdirectory, but never touch personal configuration files in your home folder, that can only be removed by hand.