SSL Troubleshooting

This page collects some tips to aid the diagnosis of SSL errors. The strategy is to test the required components with an alternative SSL implementation in a process of elimination to identify the fault.

Bear in mind that this process is not guaranteed to identify the problem if the interaction between two specific components is responsible for the problem.

We also explain some of the most common error messages you may see in logs.

Check SSL support in Erlang

The first requirement for establishing SSL connections to the broker is SSL support in the broker. Confirm that the Erlang VM has support for SSL by running the erl (or werl.exe on Windows) shell and entering

ssl:versions().
The output should look similar to this (with version number differences possible):
[{ssl_app,"5.0"},
 {supported,[tlsv1,sslv3]},
 {available,[tlsv1,sslv3]}]

If you receive an error instead, confirm that Erlang was built with OpenSSL. On Debian-based systems you may need to install the erlang-ssl package, and on Mac OS you may need to port install erlang +ssl.

Check keys and certificates with OpenSSL

We will now verify the certificates and keys specified in the configuration file against an alternative SSL implementation. This example uses OpenSSL s_client and s_server. We will confirm that the certificates and keys can be used to establish a secure link by connecting two terminals. For the examples that follow, we will assume you have the following:

Item Location
CA certificate testca/cacert.pem
Server certificate server/cert.pem
Server key server/key.pem
Client certificate client/cert.pem
Client key client/key.pem

In one terminal window execute the following command:

openssl s_server -accept 8443 -cert server/cert.pem -key server/key.pem \
  -CAfile testca/cacert.pem
In another terminal window execute
openssl s_client -connect localhost:8443 -cert client/cert.pem -key client/key.pem \
  -CAfile testca/cacert.pem
If the certificates and keys have been correctly created, an SSL connection establishment sequence will appear and the terminals will be linked. Input from either terminal will appear on the other. If the trust chain could be established, the second terminal will display this confirmation:
Verify return code: 0 (ok)

If you receive an error, confirm that the certificates and keys were generated correctly.

Check broker is listening

This step checks that the broker is listening on the expected AMQPS port. When you start the broker with a valid SSL configuration file, the broker will report the SSL listening address in the logfile. You should see an entry similar to this:

=INFO REPORT==== 8-Aug-2011::11:51:47 ===
started SSL Listener on 0.0.0.0:5671
If you included an "ssl_listeners" configuration directive and you don't see this message, it is possible that your configuration file was not read by the broker. Confirm that the configuration file quoted in the broker log contains SSL configuration options. See the configuration page for details.

Attempt SSL connection to broker

Once you have a RabbitMQ broker listening on an SSL port you can again use the OpenSSL s_client to verify SSL connection establishment, this time against the broker. This check establishes whether the broker is likely to be configured correctly, without needing to configure an AMQPS client. The example assumes a broker with an "ssl_listeners" configuration directive set to listen for SSL connections on localhost port 5671:

openssl s_client -connect localhost:5671 -cert client/cert.pem -key client/key.pem \
  -CAfile testca/cacert.pem
The output should appear similar to the case where port 8443 was used. The broker logfile should contain a new entry when the connection is established:
=INFO REPORT==== 8-Aug-2011::11:55:13 ===
accepting AMQP connection <0.223.0> (127.0.0.1:58954 -> 127.0.0.1:5671)

It should now be possible to present the broker with an AMQP connection establishment sequence over the SSL connection. If you present the broker with eight random bytes, the broker will respond with the string "AMQP" followed by an encoded version number. If you recognise the "AMQP" string, you can be confident that you are connected to an AMQP broker.

Validate client connections with stunnel

The final check is to validate AMQPS clients. We will use stunnel to provide SSL capability. In this configuration AMQPS clients will make a secure connection to stunnel, which will pass the decrypted data through to the AMQP port of the broker. This provides some confidence that the client SSL configuration is correct independently of the broker SSL configuration.

stunnel will run in daemon mode on the same host as the broker. In the discussion that follows it is assumed that stunnel will only be used temporarily. It is of course possible to use stunnel to provide SSL capability more permanently, but the lack of integration with the broker means that management reporting features and authentication plugins that use SSL information will not be able to do so.

In this example, stunnel will connect to the unencrypted AMQP port of the broker (5672) and accept AMQPS connections from SSL-capable clients on port 5679:

cat client/key.pem client/cert.pem > client/key-cert.pem
stunnel -r localhost:5672 -d 5679 -f -p client/key-cert.pem -D 7
stunnel requires a certificate and corresponding key. The generated client certificate and corresponding key should be used and concatenated as shown above with the cat command. stunnel requires that the key not be password-protected.

SSL-capable clients should now be able to connect to port 5679 and any SSL errors will appear on the console where stunnel was started.

Connect client and broker

If none of the previous steps produced errors then you can confidently connect the tested AMQPS client to the AMQPS port of the broker, making sure to stop any running OpenSSL or stunnel sessions first.

Understanding SSL logs

New broker logfile entries will be generated during many of the preceding steps. These entries together with diagnostic output from commands on the console should help to identify the cause of SSL errors. What follows is a list of the most common error entries:

Entries containing {ssl_upgrade_error, ekeyfile} or {ssl_upgrade_error, ecertfile}

This means the broker keyfile or certificate file is invalid. Confirm that the keyfile matches the certificate and that both are in PEM format. PEM format is a printable encoding with recognisable delimiters. The certificate will start and end with -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- respectively. The keyfile will likewise start and end with -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- respectively.

Entries containing {ssl_upgrade_failure, ... certify ...}

This error is related to client verification. The client is presenting an invalid certificate or no certificate. If the ssl_options has the verify option set to verify_peer then try using the value verify_none temporarily. Ensure that the client certificate has been generated correctly, and that the client is presenting the correct certificate.

Entries containing {ssl_upgrade_error, ...}

This is a generic error that could have many causes. Make sure you are using the recommended version of Erlang.