Your trip the Genius bar was too focused in my opinion. I suggest you make another appointment and lay out your case above, but this time, don't mention the memory tests, but that the problems showed up when you upgraded the RAM. Refer them to your old case number (tack this visit on as a follow-up).
Tell the Genius on hand that the problems still exist and that they need to do further testing. As it stands, your Mac is unusable. Additionally, tell them to make note that you have tried a fresh re-installation and it did nothing to help (don't mention that things ran fine for a day or too, stick to the current state of the system), which will rule out any software problems.
Ask them on what they suggest and follow it (no matter how annoying it may be). If it doesn't fix it, call them and do another follow-up. If they tested the RAM and it passes, then it might be the logicboard.
In any case, do not visit them with "the solution." Let them find it through there own testing. Clearly, the logs containing kernel panics and all the problems you list will support your case. Moreover, their internal testing tools are much more sophisticated than what you'll scrounge up on the net through 3rd party apps (they actually have in-house diagnostic utilities for these occasions).
I would highly recommend that you stop spending your time diagnosing the problem and let them do it for you. That's what they're there for.
If after everything is said and done, and they still can't pinpoint a problem, you will likely walk away with a new Mac. Just focus on the problems you are suffering from and leave the diagnostics to the Geniuses and technicians.
You may want to downgrade back to the 4 GB for now. And it's of note that replacing the RAM and the hard drive yourself do not void your warranty for that particular Mac model.
PS: You don't specify if you took your notebook to an Apple Store and spoke with a Genius or some Apple authorized repair centre. I would strongly encourage you to deal with Apple directly. They'll not only treat you better, but likely solve the problem with less headache. My advice is tailored to them; you are totally on your own should you choose a 3rd party repair shop. To galvanize the need to deal with Apple, let me put forth the following. I had a friend who purchased a Mac mini. There were strange problems during boot up where it would occasionally load the keyboard set-up assistant. The machine was also sluggish and exhibited other unusual anomalies in its day to day. We took it into an Apple Store and the Genius tried to reproduce the errors. He couldn't. He also expressly stated that minis never required repair or servicing. But with all that, he elected to replace the entire unit and gave my friend a new mini. Took it back to his place and it hummed, no more issues. I doubt he would have received the same treatment from a computer store authorized to service Macs. Deal with Apple direct; I cannot stress that enough.
My experience with inactive memory is similar to yours and as soon as free memory approaches zero, performance takes a big dive. An SSD makes a big difference, not because of RAM usage or even because it improves paging speed, but because it makes it very fast to restart a program and release the "extra" memory it's using. I don't use Photoshop and VMware but Firefox, Safari, and Chrome are all memory hungry and don't appear to ever release memory once they grab it so restarting any of them usually is a quick fix.
Memory usage in Chrome is not as obvious as others because Chrome runs as many separate processes so you don't see a single big number for memory usage.
Best Answer
It could be pure out of memory, but it may be that both products are fighting over the OpenGL implementation that Adobe uses.
Try this - it may need some fiddling to get the best balance
[this is for Photoshop, idk the precise locations for other Adobe apps]
In Prefs > Performance, reduce the overall memory requirement - it ought to handle this OK by itself, running into pagefile if needed, but just to see how it goes with this reduced…
Either disable "Use Graphics Processor" or open Advanced Settings… & selectively disable sections until you find which is causing the issue.
If you decide to leave "Use Graphics Processor" on, then look in Prefs > 3D & drop the memory requirement in there to half your card's total memory, less a little for the system.
All these tweaks will require you to relaunch each app, so it may take some time to finally settle at something that works best for you.
The alternative, as you first surmised, would be to get a more powerful graphics card with greater memory - if that would be even possible on a Macbook, which I somehow doubt.
Everymac states [for the high-end i7 machine]:
One thing you should definitely check is that it is at least switching over to the NVidia rather than using the Intel card.
gfxCardStatus would be the go-to tool for that kind of task.