Launchd starting a new process

That isn't actually launchctl creating a new process; it's most likely dnsmasq itself. If it fork()s into the background, launchd can't tell; then when it talks to launchd, launchd creates an anonymous entry for it. Make sure dnsmasq doesn't background itself.

(The - listed for the regular launchctl list output indicates that the original dnsmasq process exited.)

First, is this being opened by the system launchd (process ID 1, running as root), or a user launchd (running as a user)? If it's the system launchd, you're looking for a LaunchDaemon; if it's a user launchd, you're looking for a LaunchAgent.

Assuming it's the system launchd, you can look for the relevant LaunchDaemon like this:

grep 5901 {,/System}/Library/LaunchDaemons/*

(note: if port 5901 were listed in /etc/services, you'd also have to look for it by name as well as number. But it isn't, so you don't have to worry about that.)

It's also possible that it's a LaunchDaemon that was loaded from some other place; tracking that down would be messy, so start with the easy check first.