MacOS – How to determine where a process is being started from, and disable it

That isn't actually launchctl creating a new process; it's most likely dnsmasq itself. If it fork()s into the background, launchd can't tell; then when it talks to launchd, launchd creates an anonymous entry for it. Make sure dnsmasq doesn't background itself.

(The - listed for the regular launchctl list output indicates that the original dnsmasq process exited.)

There is a command execsnoop that you can run in a terminal window. Like so:

sudo execsnoop -v

A more detailed log of all that is going on requires a bit of dtrace hacking, as you need to track the fork and _exit system calls as well.

Edited to add a bit of explanation: Processes don't “start” on a unix system. New processes are created by the fork system call, which results in the calling process being split into two (almost) identical processes. One (the parent) keeps its PID, while the other (the childe) given a new PID. The most common reason for a fork is for the child to exec a new program; it's this fork+exec combination that you most commonly think of as a new process starting up. This is why you need to track three system calls (fork, exec, _exit) for the complete view. But just tracking exec, as execsnoop will do, seems sufficient for your purposes.