MacBook – connect 3 display to a mid 2012 MBP retina

Some displays allow you to specify the DisplayPort version. If your display defaults to DisplayPort 1.1, changing it to DisplayPort 1.2 via the OSD might improve the DisplayPort connection.

So it looks like you have systematically ruled out all hardware issues on all components, so you're right and it sounds software related. And since you were using at least one of the monitors before, it's likely not a compatibility issue.

Just in case, I would connect an external monitor (even though it appears unusable) and then check for software updates in the App Store - maybe there's a necessary firmware update to install that has yet to be triggered.

What I would do next is to test the issue in a new user account - if it happens there, it's a system-wide issue. If it doesn't, the problem could be with a corrupt system preference file somewhere in your user library that needs to be regenerated. If you have a backup from before this issue occurred, restoring your user's home folder from the backup would actually be the quickest and easiest solution to this problem, as hunting down individual preference files can be tedious when you don't know the exact one causing the issue.

If it's a system-wide issue, it's likely OS related not system library .plist related as you still had the same issue in safe mode. (Although if the monitors are identical it could be a driver issue and I would check the manufacturer's website for new drivers, but if they're different and both are not working that is highly unlikely.)

Next I would back up your system and then reinstall the OS from the App Store to see if that fixes it. I know that's time consuming but I usually set mine up before I go to bed and it's done long before I wake up.

Last but not least is an Erase and Install from the Recovery Partition, but let's hope it doesn't come to that!