Your trip the Genius bar was too focused in my opinion. I suggest you make another appointment and lay out your case above, but this time, don't mention the memory tests, but that the problems showed up when you upgraded the RAM. Refer them to your old case number (tack this visit on as a follow-up).
Tell the Genius on hand that the problems still exist and that they need to do further testing. As it stands, your Mac is unusable. Additionally, tell them to make note that you have tried a fresh re-installation and it did nothing to help (don't mention that things ran fine for a day or too, stick to the current state of the system), which will rule out any software problems.
Ask them on what they suggest and follow it (no matter how annoying it may be). If it doesn't fix it, call them and do another follow-up. If they tested the RAM and it passes, then it might be the logicboard.
In any case, do not visit them with "the solution." Let them find it through there own testing. Clearly, the logs containing kernel panics and all the problems you list will support your case. Moreover, their internal testing tools are much more sophisticated than what you'll scrounge up on the net through 3rd party apps (they actually have in-house diagnostic utilities for these occasions).
I would highly recommend that you stop spending your time diagnosing the problem and let them do it for you. That's what they're there for.
If after everything is said and done, and they still can't pinpoint a problem, you will likely walk away with a new Mac. Just focus on the problems you are suffering from and leave the diagnostics to the Geniuses and technicians.
You may want to downgrade back to the 4 GB for now. And it's of note that replacing the RAM and the hard drive yourself do not void your warranty for that particular Mac model.
PS: You don't specify if you took your notebook to an Apple Store and spoke with a Genius or some Apple authorized repair centre. I would strongly encourage you to deal with Apple directly. They'll not only treat you better, but likely solve the problem with less headache. My advice is tailored to them; you are totally on your own should you choose a 3rd party repair shop. To galvanize the need to deal with Apple, let me put forth the following. I had a friend who purchased a Mac mini. There were strange problems during boot up where it would occasionally load the keyboard set-up assistant. The machine was also sluggish and exhibited other unusual anomalies in its day to day. We took it into an Apple Store and the Genius tried to reproduce the errors. He couldn't. He also expressly stated that minis never required repair or servicing. But with all that, he elected to replace the entire unit and gave my friend a new mini. Took it back to his place and it hummed, no more issues. I doubt he would have received the same treatment from a computer store authorized to service Macs. Deal with Apple direct; I cannot stress that enough.
Because OSX manages memory (as does Windows) it is often difficult to tell if performance issues are caused by memory limits. Technically, you really want the OS to capture all the available memory and allocate as needed, in which case, you would see all of the memory being 'used', even though there may be memory available but not allocated by the OS.
This is somewhat analogous to what is happening in OSX and Win7+. However, Apple has made it confusing in the past with many memory classifications, like Wired, Swap, Real, etc.
Luckily, you are running Mavericks, and a new feature of Mavericks is the Memory Pressure graph, described in the Apple support document here: http://support.apple.com/kb/HT5890
Basically, if the memory graph shows Red, you need more RAM, if its Green, you are good. I suggest reading the article for more details, but the Red, Yellow, Green is a great visual tool to help guide where you might be seeing problems.
Depending on your model, I have personally found Macs suffer more from hard drive access speed than memory limits, so an SSD, if your iMac does not have one, has a huge impact (and unfortunately quite a bit of work to put into your iMac)
Best Answer
Looks like the 560 massively outperforms the 555 in benchmark testing:
http://gpuboss.com/gpus/Radeon-RX-560-vs-Radeon-Pro-555
And looks like it has twice the VRAM (4 Gb vs 2 Gb) which to me makes it a pretty easy decision.
FWIW - I'm a graphically-intensive applications user, and having a GPU with more VRAM means better performance from apps which target GPU processing, so I'm not a typical use-case.