The problem is that certain apps don't provide a title for the system tray icons. Windows doesn't have a way of recognizing them if they don't have a title, so it doesn't remember the option to always hide. That's why certain icons just can't remember to stay hidden.
The system-related menubar items (Sync, BLuetooth, Wireless, Time Machine, Sound, Keyboard, Time, Battery, etc.) and very few others from 3rd party applications can be removed from the Menu Bar by Drag&Drop while pressing Cmd. Be aware that you e.g. have no way to start a Time Machine backup anywhere else but this menu.
Many applications allow you to configure whether they show menubar icons, for example (in the screenshot) Adium (Preferences, General, uncheck "Show Adium status in menu bar").
Some applications (e.g. Time Sink by Manytricks) leave you a choice between menubar icon or Dock icon: It's your choice which is better.
Yet others require menubar icons, e.g. f.lux, Dropbox (also in the screenshot), or Caffeine; primarily applications without Dock icon. You can only open the application bundle and try to find the icons in there, and replace them with an empty icon. For example, I don't need to see the "Caffeine is running" icon, so I can replace the icon of the coffee cup with coffee in it with an empty image. This will not "remove" the icon from the menu bar, merely make it invisible (you be the judge if this is for you).
You can remove the Spotlight menubar icon by following this hint. You can remove the icon and still have spotlight functionality via an application launcher such as LaunchBar, or any finder window's search box.
The shareware program Bartender can hide most notification area icons.
Best Answer
PS Tray Factory can hide icons in the system tray. It's not free though.