MacOS – How to get rid of “application downloaded from the internet” message when installing homebrew casks

I see a few issues from what you posted.

error: unable to unlink old 'Library/Homebrew/test/mach/a.out' (Permission denied)

It seems to me you have a permissions issue perhaps from the leftover previous installation of Homebrew. You should remove all traces of the previous installation before you reinstall.

And Homebrew doesn't want to be installed as root

Why on earth would you want to install homebrew as root? The whole point of the project is to keep homebrew in /usr/local so that you don't need sudo.

Warning: It appears you have MacPorts or Fink installed. Software installed with other package managers causes known problems for Homebrew.

You should also ensure MacPorts is uninstalled and that you've uninstalled Fink. They could cause issues with software installed by homebrew.

Warning: An outdated version of Git was detected in your PATH. Git 1.7.10 or newer is required to perform checkouts over HTTPS from GitHub.

This indicates to me that you still have an old version of Git in your PATH, probably installed with another package manager. OS X Mavericks comes with Git version 1.8.5.1

What is the result of the whereis git command from the Terminal? usr/bin/git is the OS X install of Git.

Follow the guides to completely remove previous MacPorts or Fink installs and ensure the old directories are removed from your PATH variable.

PS.

I'll be sad if I can't use Mac OS X as a hacker's Unix

That would make me sad too. :(

I find homebrew invaluable and easy to work with. In summary, it seems you just have old software installed (and perhaps extraneous directories in your PATH) and need to clear it out, check your permissions, and reinstall homebrew from scratch.

Short term answer, try this command to see if your brew installation worked:

/usr/local/bin/brew doctor

It seems that your bash environment may not be looking in the /usr/local/bin for executibles (which means you have to write the full path rather than just the name of the command). The solution may be just resourcing your conf files, which can be roughly done by closing your terminal and opening a new one. Otherwise, you'll want to add a line to your ~/.bash_profile. I think resourcing will do the trick though.

So after confirming the success with the first doctor, try quitting your terminal application and starting fresh, then your original usage of brew should work. If it doesn't try:

echo $PATH

and report back the results.