MacOS – Brew cask: I would like to check if the package/formula is already installed through brew or other means

Take the path that you used for --prefix= during your ./configure and sym-link that into the Cellar/ directory that Homebrew is using. Homebrew will see these directories and see the dependencies as satisfied (unless you need to have them compiled with or without certain features to make Homebrew's formulas compile correctly). If your manually-compiled deps aren't compiled correctly, you'll probably get errors out of Homebrew.

ln -s /path/from/prefix/ /usr/local/Cellar/path/from/prefix

(Note the trailing slash on the first path, and lack of trailing slash on the second. OS X can be picky about these slashes when creating sym-links.)

This is the wrong way to do it, you should just be using --ignore-dependencies coq — that's what it's there for.

This command should show you where any openssl binaries may be located on your system:

locate openssl | grep bin/openssl

For homebrew, make sure you have /usr/local/bin and /usr/local/sbin in your $PATH. You can check your current path setting by issuing echo $PATH.