Sadly, malicious code can be so small and obfuscated in an infinite number of ways so you can't simply make a rule to test a package to tell if it's good or bad.
Statistically, people have established some guidelines like trying to know the source (typing in the web address directly and not trusting a link, not downloading it from an alternate site, insisting that it be code-signed and verifying the package check sum before installing executables), but these are indirect remedies.
Apple maintains a page with links to most security information here:
Additionally, you could insist that all software that you run is signed by a legitimate developer to reduce the likelihood that someone has added a key logger to the game you just downloaded.
Lastly, after several delays, Apple has released GateKeeper to effectively sandbox software so that it has to explicitly list the things that it does (like access the address book, or modify files without user interaction) so that you can only let the system run code that obeys the sandbox rules. From a consumer perspective, you can prevent all code from running while you do whatever checks you feel are appropriate before allowing that code to run on your Mac.
Best Answer
Malwarebytes is at best terribly ineffective from my experience. Why do you suspect a rootkit? I don't think you're wrong, possibly very perceptive but just wondering why.
The problem with rootkits is they will hide all evidence of their existence especially from a rickety Malwarebytes scan. Mac malware creators must be sophisticated enough to jump some of the small hurdles Apple attempts to create so keep that in mind in terms of the persistence factor at play.
What makes you think that the malware won't go hide in your NVRAM, xartstorage (secure enclave, graphics card, SMC, create a RAMDisk, etc. to make you think that it's gone and that Recovery Mode really is an almighty kill switch and isn't just a pacifier that effectively negates all legitimate and acceptable levels suspicion.
macOS (previously styled OS X) is inherently insecure. "It just works" is not the befitting of a truly secure operating system. In which case "I expletive hate it," would probably be more likely. Focus groups probably would have found that didn't go have quite the same cachet.
To assess the potential of this type of malware: