MacOS – What’s the best way to update Homebrew when upgrading macOS

I see a few issues from what you posted.

error: unable to unlink old 'Library/Homebrew/test/mach/a.out' (Permission denied)

It seems to me you have a permissions issue perhaps from the leftover previous installation of Homebrew. You should remove all traces of the previous installation before you reinstall.

And Homebrew doesn't want to be installed as root

Why on earth would you want to install homebrew as root? The whole point of the project is to keep homebrew in /usr/local so that you don't need sudo.

Warning: It appears you have MacPorts or Fink installed. Software installed with other package managers causes known problems for Homebrew.

You should also ensure MacPorts is uninstalled and that you've uninstalled Fink. They could cause issues with software installed by homebrew.

Warning: An outdated version of Git was detected in your PATH. Git 1.7.10 or newer is required to perform checkouts over HTTPS from GitHub.

This indicates to me that you still have an old version of Git in your PATH, probably installed with another package manager. OS X Mavericks comes with Git version 1.8.5.1

What is the result of the whereis git command from the Terminal? usr/bin/git is the OS X install of Git.

Follow the guides to completely remove previous MacPorts or Fink installs and ensure the old directories are removed from your PATH variable.

PS.

I'll be sad if I can't use Mac OS X as a hacker's Unix

That would make me sad too. :(

I find homebrew invaluable and easy to work with. In summary, it seems you just have old software installed (and perhaps extraneous directories in your PATH) and need to clear it out, check your permissions, and reinstall homebrew from scratch.

If the path includes spaces, you're going to need to quote it.

Your line export PATH=/Users/ps1/Documents/PS 1/whatever-else/:$PATH causes Bash to interpret the PATH=/Users/ps1/Documents/PS part as the full first argument to your PATH—telling export to set the PATH to only the section before the space. If you use export "PATH=/Users/ps1/Documents/PS 1/whatever-else/:$PATH", it will interpret the entire PATH=/Users/ps1/Documents/PS 1/whatever-else/:$PATH as part of the first argument. By convention, most people only quote the right side of the equals sign, because it looks nicer, it's the only thing that needs quoted, and bash automatically concatenates strings with no spaces between them.