Well, it looks like I might finally have fixed it without losing all my settings/data. But I warn you, it isn't pretty and it involved temporarily jailbreaking my phone.
First I jailbroke my phone, I used the blackra1n app, but any jailbreaking technique would work equally well.
Then I used cydia to install the OpenSSH package (again any jailbrake package installed would do).
Then I ssh'd into the phone (username = root, password = alpine).
I cd'd into /private/var/mobile/library/safeharbor and took a copy of the directory listing, so I had a note of which apps I had installed. This directory is where all the data for the apps is stored.
I then deleted the file /var/mobile/Library/Caches/com.apple.mobile.installation.plist, this is the file that keeps details of which apps are install. I then rebooted the phone.
At this point the only apps showing as installed were the Apple supplied apps, cydia and blackra1n.
I then backed up the phone. I then restored to the current firmware and let iTunes restore the backup I had just made.
So now I have a clean phone with no apps installed, but importantly I still have the settings and data for the apps I had installed. I have started reinstalling my apps and they are properly picking up their data and settings that are still there in the safeharbor directory.
I have checked that syncing of apps works and am just keeping my fingers crossed that it stays that way.
The important trick here was deleting the com.apple.mobile.installation.plist file, because then iTunes doesn't know that there were any apps installed and so doesn't start deleting the data/settings when it syncs.
This is pretty tricky, as there are several files containing iTunes metadata in addition to live running instances of iTunes. The trickiest of these to overcome is the fact that the running instance of iTunes does not write out data until it quits, so if you have running instances on both your iMac and MBP and you update the files underneath them, they will get lost. Further, paths are contained within the files, so this will not work unless you have identical paths on both machines (assuming you have not changed the defaults and have the same username, this would put the music in /Users/username/Music/iTunes/iTunes Music/ and the metadata in /Users/username/Music/iTunes/{iTunes Library,iTunes Music Library.xml}).
You can keep the music folder up to date on both by using rsync, but this only works cleanly if you either always add or delete items on one of the machines or if you only add items. If so, the two lines on the MBP would work:
% rsync -a /Users/<username>/Music/iTunes/iTunes\ Music <G5hostname>:/Users/<username>/Music/iTunes
% rsync -a <G5hostname>:/Users/<username>/Music/iTunes/iTunes\ Music /Users/<username>/Music/iTunes
You'll probably want to set up automatic authentication using public keys auth in SSH and put this in a cronjob.
To accomplish moving the metadata, I suggest you have a script running on both machines under cron which first determines if iTunces is running on the local machine and if it is not, it rsyncs the metadata files from the other server if the content on the other server is newer than that locally. Something like:
if [ `ps -ax | grep -v grep | grep -c iTunes` -ne 0 ] ; do
rsync -au <otherhostname>:/Users/<username>/Music/iTunes/{iTunes\ Library,iTunes Music\ Library.xml} /Users/<username>/Music/iTunes
done
This still assumes that you don't make mods both instances with both instances being up continuously. Also, I have not tested this out, so there are probably some syntax errors or subtleties I'm missing.
Best Answer
Finally, almost by accident, I figured out a way to "force" iTunes to sync those songs, and they finally showed up on my iOS device (iPhone).
The trick was to make a change in the "properties" associated with the song (or album): I discovered this by updating the artwork, but any change triggers the re-sync:
After making the change, then select the phone and hit the
Sync
button; shortly after, you will see activity on the top bar:and:
While this goes on, switching back to the view of the music uploaded to the iPhone, you can see the grayed-out songs finally being updated:
As this has been bugging me for years, I hope others will find this useful!