Ubuntu – 12.04.4 server can’t verify common SSL certificates, usual fixes failing

certificatescurlSecurityserverssl

The Problem

I've got one server in a farm which is suddenly unable to correctly handle SSL certificates. Attempting to do a curl command like curl -v https://google.com results in:

curl -v https://google.com
* About to connect() to google.com port 443 (#0)
*   Trying 74.125.137.101... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Using openssl s_client is a little more detailed.

# openssl s_client -host google.com -port 443
CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate

Things Tried So Far

Reinstalling ca-certificates – already have the latest available version, according to aptitude, Version: 20130906ubuntu0.12.04.1.
Reconfiguring ca-certificates via dpkg-reconfigure. This appears to rehash the /etc/ssl/certs folder but has no effect on the problem.
Using update-ca-certificates --fresh to regenerate the symbolic links in that folder
Grabbing the latest Mozilla ca bundle from curl.haxx.se – by putting that .pem file in /etc/ssl/certs and running the update command.

Weirdness

The certificate that curl claims it cannot find is indeed in the certification path.

# ls -l /etc/ssl/certs/*Geo*

lrwxrwxrwx 1 root root 57 Apr  7 15:57 /etc/ssl/certs/GeoTrust_Global_CA.pem -> /usr/share/ca-certificates/mozilla/GeoTrust_Global_CA.crt
...

The certificate file referenced has the same permissions as every other box on my network, namely 644.

# ls -l /usr/share/ca-certificates/mozilla/GeoTrust_Global_CA.crt
-rw-r--r-- 1 root root 1216 Feb 20 11:49 /usr/share/ca-certificates/mozilla/GeoTrust_Global_CA.crt

Other secure sites such as Github show identical issues with different certificates. I am running the absolute latest version available of all packages for Ubuntu 12.04.4, including curl, openssl, and ca-certificates.

What's going on here?

Best Answer

I also have this problem, try this:

openssl s_client -host google.com -port 443

this command will also print a cert chain, the last one is:

s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

so you also need https://www.geotrust.com/resources/root_certificates/certificates/Equifax_Secure_Certificate_Authority.pem.

This cert path is different with cert path showed in browser (IE, Firefox, Chrome), I don't know why, but this fix my problem.

Related Solutions

Ubuntu – OpenSSL not picking up CAs in certs folder

There are several cryptographic libraries on your system:

OpenSSL (the gold standard, with a BSD-style (very free) licence that includes a somewhat problematic clause (preventing GPL compatibility, but nothing “bad”) limiting its adoption in the GNU world)
GnuTLS (the replacement from the FSF; comes in two flavours, LGPLv2-licenced (but unmaintained) and LGPLv3-licenced (and thus incompatible with GPLv2-only programs); historically not as featureful as OpenSSL, a bit more buggy, but more strict too, which enhances security)
NSS (Netscape/Mozilla's library, rarely used outside; slow to adopt new standards)
minor ones like PolarSSL, MatrixSSL, NaCl/Salt

All of them have, of course, similarities and differences. Software that uses them (for cryptographic purposes, or to use SSL/TLS) sometimes supports using more than one of these libraries (for example Lynx, the web browser, is normally linked against OpenSSL but supports GnuTLS too (just not as good) in order to appease the GNU people).

cURL also is one of the projects supporting using either of the three major crypto libraries. This is mostly because cURL is, primary, a library intended to be used by yet other programs when they want to download (or even upload) things using http, ftp, etc. connections. The curl command-line tool can come from either of these variants.

Now, I'm fairly sure that the problem you're seeing with the not-freshly-installed system is the following:

OpenSSL and GnuTLS both support using /etc/ssl/certs/<hash>.<number>-style CA directories. OpenSSL version 0.x and GnuTLS however use a different algorithm to calculate the aforementioned hash than OpenSSL version 1.x uses. (Both can coëxist on a system; if different certificates have the same hash you just use a differing number for them. But for some reason, Debian/Ubuntu's ca-certificates package doesn't seem to do this.) Additionally, some versions of GnuTLS did not support using the directory, but only using a file /etc/ssl/certs/ca-certificates.crt (which is also usually managed by the ca-certificates package's maintainer scripts, but can deviate); you seem to be using an older version, so this may be the thing you hit.

openssl s_client by default (i.e. without the -CApath or -CAfile option) does not look anywhere for certificates.

Your curl from the upgraded installation most likely uses a different crypto library than the curl from the fresh installation.

Try openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -connect the-problem-site.com:443 in addition to openssl s_client -CApath /etc/ssl/certs -connect the-problem-site.com:443 to mimic the behaviour of older GnuTLS versions.

Double-check if there's an OpenSSL 1.x anywhere on your system (Ubuntu is known for sneaking major updates even into LTS versions), and if yes, check the hash of the file:

openssl x509 -noout -hash -in /etc/ssl/certs/GeoTrust_Global_CA.pem
openssl x509 -noout -subject_hash_old -in /etc/ssl/certs/GeoTrust_Global_CA.pem
openssl x509 -noout -subject_hash -in /etc/ssl/certs/GeoTrust_Global_CA.pem

Normally, either, the second and third command should fail (OpenSSL 0.x), or the first and third command should display the same hash but the second one should display a different hash (OpenSSL 1.x). GnuTLS would use the output from the second command (if OpenSSL 1.x is installed); if OpenSSL 0.x is installed that's the same hash. You can create such symlinks manually.

I can update this posting once you provide debugging feedback.