kstart --iconify <program name> in KDE starts minimized. I really don't know if there is a specific command to close to systray (even because many apps don't have a systray).
Another possibility is using ksystray. For example ksystraycmd --hidden kmail will start kmail and put it in the system tray. But i'm not quite sure if it will work for any app.
If not, probably you have to check if there is a specific option for your application.