I've looked around a lot to try to find the answer to this but have thus far been unsuccessful. I am programming in Python, and I have code that takes a long time to run (hours to months depending on the application) and I am trying to optimize it. On larger thread systems (once I get above ≈8 threads, but especially >18 threads), I run into a situation where the system is using up a large amount of my CPU instead of the actual code I want to run. On a 12-thread machine as I type this, the system is using ≈25–30% of my total CPU. If I try to run it on a 36-thread machine, the system takes >70% of the CPU which is simply unworkable (though running two instances of the code, limiting each to 18 threads, cuts this overhead back, oddly enough).
The Activity Monitor indicates that taskgated is using at least 10%, while notifyd, logd, and launchd are also using several percent, each (along with mds at 1% so I should turn off Spotlight, and sometimes lsd spikes to >40%, but that's more rare … note that that process is another launch system daemon).
My older hypothesis was that it was a disk I/O issue, since the code was writing and reading many small files to try to keep track of certain things and recover if I needed to stop it.
My now-working hypothesis, based on what I could find online about the interactions between taskgated and launchd, is that this particular chunk of code is spawning a large number of processes and macOS's launching and security daemons are taking large amounts of CPU to make sure those processes are safe. These are things like calling "mv" and "rm" from the command line (os.system(…) in my Python code), and spawning other code that take a second or two to run (such as in a different conda environment when two have conflicting installs of necessary components). I count at least 40 potential spots where this chunk of code is possibly spawning child processes, and I thread it so that it's doing that concurrently however many threads there are (so, on a 12-thread machine, 12×40 over the course of ~10 seconds). It doesn't seem to me like this should be making my system take up so much CPU, but that's my best guess at the moment.
Possibly related, the taskgated is constantly spitting out to log files, "MacOS error: -67062", which again I've searched for and found no luck with diagnosing the issue. And, diskarbitrationd is spawning a lot of "<private>" messages in the Console, but its CPU% is around 0.3% so I'm less concerned about that.
My apologies for rambling a bit here, but I'm trying to provide the information I have, and hopefully someone here has an idea. If I can get rid of this 25% or larger issue, that can save months of time.
For what it's worth, I'm running macOS 10.14.5 and ..4 on two desktops, and 10.15.5 on a laptop. Same issue for all. Running in Linux on a nearly identical 36-thread system build does not have the overhead issue (but I really don't want to switch to Linux) which is another reason why I don't think it's a disk I/O problem.
Best Answer
I think the overhead you are getting with the daemons you are referring to are unavoidable on macOS. For example,
launchdis the main process for launching applications and ensures that the processes it launched are kept alive if instructed so. Using more threads on macOS is a well known issue concerning a higher overhead for the kernel. This is why Apple documentation clearly states that you should use them judiciously and sparingly. Moreover, it seems as though macOS is not trusting your script, as is an unsigned “executable” and the error you are getting corresponds to:Thus, the extra overhead you are seeing is most probably due to Gatekeeper, which is constantly checking what your script is spawning and doing.
Possible (partial) solutions to your problem: