So you’ve configured Apache just like you want to. You’ve even added an SSL certificate to allow your users to securely navigate on your site. And you test it, and everything works, across all browsers you have access to… and you receive a dreaded call that there is a certificate error on a client (typically some higher up, because that is just your luck). What the hey?

Assuming you have tried all the simple debugging steps, ask to see the bug in person. And look for one thing only: is the certificate offered by the server the one you’d expect?

This was particularly annoying to find, so I’ll explain a bit better. Go to your computer where everything works so nicely, and use a decent browser to go to the SSL version of it. Click on the locker icon and view the certificate. Head on to the advanced details.

For Chrome users, that would be by clicking on the green locker, going to the Connection tab, and clicking on “Certificate Information”. Then go to the “Details” tab, and make a note of the “Serial Number” entry. Go through the same process on the client browser ( IE, which is likely what the person is using, will prompt to view the certificate when the problem arises ). Compare the serial.

If you have my kind of luck, the serials won’t match.

Even worse, in my case this is a brand new Linux machine, with just one vhost – how is there a different certificate installed? The answer is hidden in the docs for mod_ssl FAQs :

Why can’t I use SSL with name-based/non-IP-based virtual hosts?

The reason is very technical, and a somewhat “chicken and egg” problem. The SSL protocol layer stays below the HTTP protocol layer and encapsulates HTTP. When an SSL connection (HTTPS) is established Apache/mod_ssl has to negotiate the SSL protocol parameters with the client. For this, mod_ssl has to consult the configuration of the virtual server (for instance it has to look for the cipher suite, the server certificate, etc.). But in order to go to the correct virtual server Apache has to know the Host HTTP header field. To do this, the HTTP request header has to be read. This cannot be done before the SSL handshake is finished, but the information is needed in order to complete the SSL handshake phase. See the next question for how to circumvent this issue.

In plain English? Apache cannot tell which vhost this connection is for, until AFTER SSL is established. Thus, it uses the DEFAULT settings to establish the connection, and THEN processes your vhost-specific items.

Obviously, if you have one IP per vhost this shouldn’t happen (that’s why it is the recommended approach) – but as said before, I only have ONE vhost, and ONE ip, so … ?

The answer is that when you install the mod_ssl apache module, it also creates its configuration file ( located at /etc/httpd/conf.d/ssl.conf on CentOS 6.4 ) which defines a DEFAULT configuration for ALL IPs, and since it’s processed BEFORE your custom vhost config file, it takes precedence. Guess that I found in that config? Oh yes, a self-signed certificate.

But wait! How does it work just fine on your end, and others get stumped on this issue?

The answer here is how SSL is established. This is by no means a new problem – so a solution was created, by the name of Server Name Identification (SNI). This basically provides a hint to the server as to which vhosts it is interested in BEFORE establishing the SSL connection, thus allowing the server to pick the right certificate. Looking at the compatible browsers though, one line stands out like a sore thumb:

Internet Explorer 7.0 or later (on Vista, not XP)

There’s your culprit. The solution? Just comment out the default vhost definition in ssl.conf. It really shouldn’t be there in the first place, to be honest, but to be fair? If your vhost definitions are not like this

<VirtualHost *:443>

but like this instead:

<VirtualHost 1.2.3.4:443>

as they should, your wouldn’t stump on this issue. You wouldn’t learn about all this either though, so I’m going to count this whole event as “experience” to feel better about spending all that time on something so simple.

So, your final ssl.conf should look like this (there are more changes that I have made for apache 2.4 to make the SSL more secure):

#
# When we also provide SSL we have to listen to the
# the HTTPS port in addition.
#
Listen 443 https

##
## SSL Global Context
##
## All SSL configuration in this context applies both to
## the main server and all SSL-enabled virtual hosts.
##

# Pass Phrase Dialog:
# Configure the pass phrase gathering process.
# The filtering dialog program (`builtin’ is a internal
# terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog

# Inter-Process Session Cache:
# Configure the SSL Session Cache: First the mechanism
# to use and second the expiring timeout (in seconds).
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300

# Pseudo Random Number Generator (PRNG):
# Configure one or more sources to seed the PRNG of the
# SSL library. The seed data should be of good random quality.
# WARNING! On some platforms /dev/random blocks if not enough entropy
# is available. This means you then cannot use the /dev/random device
# because it would lead to very long connection times (as long as
# it requires to make more entropy available). But usually those
# platforms additionally provide a /dev/urandom device which doesn’t
# block. So, if available, use this one instead. Read the mod_ssl User
# Manual for more details.
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
#SSLRandomSeed startup file:/dev/random 512
#SSLRandomSeed connect file:/dev/random 512
#SSLRandomSeed connect file:/dev/urandom 512

#
# Use “SSLCryptoDevice” to enable any supported hardware
# accelerators. Use “openssl engine -v” to list supported
# engine names. NOTE: If you enable an accelerator and the
# server does not start, consult the error logs and ensure
# your accelerator is functioning properly.
#
SSLCryptoDevice builtin
#SSLCryptoDevice ubsec

##
## SSL Virtual Host Context
##

#<VirtualHost _default_:443>

# General setup for the virtual host, inherited from global configuration
#DocumentRoot “/var/www/html”
#ServerName www.example.com:443

# Use separate log files for the SSL virtual host; note that LogLevel
# is not inherited from httpd.conf.
#ErrorLog logs/ssl_error_log
#TransferLog logs/ssl_access_log
#LogLevel warn

# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
#SSLEngine on

# SSL Protocol support:
# List the enable protocol levels with which clients will be able to
# connect. Disable SSLv2 access by default:
#SSLProtocol all -SSLv2 -SSLv3

# SSL Cipher Suite:
# List the ciphers that the client is permitted to negotiate.
# See the mod_ssl documentation for a complete list.
#SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA

# Speed-optimized SSL Cipher configuration:
# If speed is your main concern (on busy HTTPS servers e.g.),
# you might want to force clients to specific, performance
# optimized ciphers. In this case, prepend those ciphers
# to the SSLCipherSuite list, and enable SSLHonorCipherOrder.
# Caveat: by giving precedence to RC4-SHA and AES128-SHA
# (as in the example below), most connections will no longer
# have perfect forward secrecy – if the server’s key is
# compromised, captures of past or future traffic must be
# considered compromised, too.
#SSLCipherSuite RC4-SHA:AES128-SHA:HIGH:MEDIUM:!aNULL:!MD5
#SSLHonorCipherOrder on

# Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate. If
# the certificate is encrypted, then you will be prompted for a
# pass phrase. Note that a kill -HUP will prompt again. A new
# certificate can be generated using the genkey(1) command.
#SSLCertificateFile /etc/pki/tls/certs/localhost.crt

# Server Private Key:
# If the key is not combined with the certificate, use this
# directive to point at the key file. Keep in mind that if
# you’ve both a RSA and a DSA private key you can configure
# both in parallel (to also allow the use of DSA ciphers, etc.)
#SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

# Server Certificate Chain:
# Point SSLCertificateChainFile at a file containing the
# concatenation of PEM encoded CA certificates which form the
# certificate chain for the server certificate. Alternatively
# the referenced file can be the same as SSLCertificateFile
# when the CA certificates are directly appended to the server
# certificate for convinience.
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

# Certificate Authority (CA):
# Set the CA certificate verification path where to find CA
# certificates for client authentication or alternatively one
# huge file containing all of them (file must be PEM encoded)
#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt

# Client Authentication (Type):
# Client certificate verification type and depth. Types are
# none, optional, require and optional_no_ca. Depth is a
# number which specifies how deeply to verify the certificate
# issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth 10

# Access Control:
# With SSLRequire you can do per-directory access control based
# on arbitrary complex boolean expressions containing server
# variable checks and other lookup directives. The syntax is a
# mixture between C and Perl. See the mod_ssl documentation
# for more details.
#<Location />
#SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
# and %{SSL_CLIENT_S_DN_O} eq “Snake Oil, Ltd.” \
# and %{SSL_CLIENT_S_DN_OU} in {“Staff”, “CA”, “Dev”} \
# and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
# and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20 ) \
# or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
#</Location>

# SSL Engine Options:
# Set various options for the SSL engine.
# o FakeBasicAuth:
# Translate the client X.509 into a Basic Authorisation. This means that
# the standard Auth/DBMAuth methods can be used for access control. The
# user name is the `one line’ version of the client’s X.509 certificate.
# Note that no password is obtained from the user. Every entry in the user
# file needs this password: `xxj31ZMTZzkVA’.
# o ExportCertData:
# This exports two additional environment variables: SSL_CLIENT_CERT and
# SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
# server (always existing) and the client (only existing when client
# authentication is used). This can be used to import the certificates
# into CGI scripts.
# o StdEnvVars:
# This exports the standard SSL/TLS related `SSL_*’ environment variables.
# Per default this exportation is switched off for performance reasons,
# because the extraction step is an expensive operation and is usually
# useless for serving static content. So one usually enables the
# exportation for CGI and SSI requests only.
# o StrictRequire:
# This denies access when “SSLRequireSSL” or “SSLRequire” applied even
# under a “Satisfy any” situation, i.e. when it applies access is denied
# and no other module can change it.
# o OptRenegotiate:
# This enables optimized SSL connection renegotiation handling when SSL
# directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
#<Files ~ “\.(cgi|shtml|phtml|php3?)$”>
# SSLOptions +StdEnvVars
#</Files>
#<Directory “/var/www/cgi-bin”>
# SSLOptions +StdEnvVars
#</Directory>

# SSL Protocol Adjustments:
# The safe and default but still SSL/TLS standard compliant shutdown
# approach is that mod_ssl sends the close notify alert but doesn’t wait for
# the close notify alert from client. When you need a different shutdown
# approach you can use one of the following variables:
# o ssl-unclean-shutdown:
# This forces an unclean shutdown when the connection is closed, i.e. no
# SSL close notify alert is send or allowed to received. This violates
# the SSL/TLS standard but is needed for some brain-dead browsers. Use
# this when you receive I/O errors because of the standard approach where
# mod_ssl sends the close notify alert.
# o ssl-accurate-shutdown:
# This forces an accurate shutdown when the connection is closed, i.e. a
# SSL close notify alert is send and mod_ssl waits for the close notify
# alert of the client. This is 100% SSL/TLS standard compliant, but in
# practice often causes hanging connections with brain-dead browsers. Use
# this only for browsers where you know that their SSL implementation
# works correctly.
# Notice: Most problems of broken clients are also related to the HTTP
# keep-alive facility, so you usually additionally want to disable
# keep-alive for those clients, too. Use variable “nokeepalive” for this.
# Similarly, one has to force some clients to use HTTP/1.0 to workaround
# their broken HTTP/1.1 implementation. Use variables “downgrade-1.0” and
# “force-response-1.0” for this.
#BrowserMatch “MSIE [2-5]” \
# nokeepalive ssl-unclean-shutdown \
# downgrade-1.0 force-response-1.0

# Per-Server Logging:
# The home of a custom SSL log file. Use this when you want a
# compact non-error SSL logfile on a virtual host basis.
#CustomLog logs/ssl_request_log \
# “%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \”%r\” %b”

#</VirtualHost>

# Begin copied text
# from https://cipherli.st/
# and https://raymii.org/s/tutorials/Strong_SSL_Security_On_Apache2.html

SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3
SSLHonorCipherOrder On
# Disable preloading HSTS for now. You can use the commented out header line that includes
# the “preload” directive if you understand the implications.
#Header always set Strict-Transport-Security “max-age=63072000; includeSubdomains; preload”
Header always set Strict-Transport-Security “max-age=63072000; includeSubdomains”
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
# Requires Apache >= 2.4
SSLCompression off
SSLUseStapling on
SSLStaplingCache “shmcb:logs/stapling-cache(150000)”
# Requires Apache >= 2.4.11
# SSLSessionTickets Off