How do you sync only new songs? You put them in your library, and click sync. If you have duplicates in your library, you will have duplicates on your device. What you see listed in iTunes is exactly what you will have on your device.
The answer is not any harder than this. "Sync" means exactly what it says on the tin, it updates your device with the data in your library. It will "update" songs that have had information changed (cover art, lyrics, or if a title was corrected, etc.), and it will add new ones you've put in the iTunes library.
When you click sync, iTunes does not overwrite everything and start from scratch. It synchronizes your library. It picks up new changes and writes them to the device, and it will even bring new content back from the device if you bought something from the store on your device.
Considering your exact requirements:
Do not change any settings in iTunes. Leave music syncing options completely default, and iTunes will do exactly what you want it to do.
Your exact requirement is exactly what iTunes is supposed to do.
An iOS device's backup typically contains just applications' settings & data. So, if you have just made a backup without saving app or music, you've lost everything.
Sorry :'(
Best Answer
From what I see using Vox, it does not create a library of its own, instead it either access the iTunes Library directly for information.
It should be a read-only access, and you have nothing to worry.
You should note that it's not iTunes integration, so it does not work with iTunes Match to provide tracks that aren't downloaded to your Mac yet.
If you have further concerns, I suggest you contact the developers of Vox directly.