Found the answer for what's going on here, TL;DR:
Remove quarantine for the offending app, in my case:
sudo xattr -d -r -s com.apple.quarantine ~/Library/PreferencePanes/PTHPasteboard.prefPane
Apparently with 10.12 (Sierra) Apple has implemented something called "App Translocation" and is documented under the "What's New in OS X" -> "Security and Privacy Enhancements":
Starting in OS X v10.12, you can no longer provide external code or data alongside your code-signed app in a zip archive or unsigned disk image. An app distributed outside the Mac App Store runs from a randomized path when it is launched and so cannot access such external resources.
There is further explanation in "OS X Code Signing In Depth":
If using a disk image to ship an app, users should drag the app from
the image to its desired installation location (usually /Applications)
before launching it. This also applies to apps installed via ZIP or
other archive formats or apps downloaded to the Downloads directory:
ask the user to drag the app to /Applications and launch it from
there.
This practice avoids an attack where a validly signed app launched
from a disk image, ZIP archive, or ISO (CD/DVD) image can load
malicious code or content from untrusted locations on the same image
or archive. Starting with macOS Sierra, running a newly-downloaded app
from a disk image, archive, or the Downloads directory will cause
Gatekeeper to isolate that app at a unspecified read-only location in
the filesystem. This will prevent the app from accessing code or
content using relative paths.
See this for further information.
I really appreciate what the author of that article notes here:
Under what circumstances does App Translocation occur?
First, the app must have a com.apple.quarantine extended attribute. If
you delete the quarantine xattr, then App Translocation does not
occur, and the app will launch from where it was unarchived, like
normal.
Second, the app must be opened by Launch Services. This usually means
Finder, but it can also mean open from Terminal, for example. If you
launch the app executable directly from bash, on the other hand, App
Translocation does not occur.
Third, the app must not have been moved — by Finder. If you move the
app, using Finder, from the app's original unarchived location to
another folder, even a subfolder, e.g., ~/Downloads/Test/, then App
Translocation does not occur.
However, if you move the app using mv from Terminal, then App
Translocation will still occur. Normally you would move the app from
~/Downloads to /Applications, and that would cause the app to be
launched from /Applications like normal, but the locations of the
particular folders don't seem to matter.
The mere act of moving the app using Finder stops App Translocation
from happening. Indeed, once you've moved the app once, it will no
longer experience App Translocation again, even if you then move it
back to ~/Downloads.
Best Answer
This appears to be fixed in 10.13.4.