Menu

TLS/SSL Troubleshooting

Introduction

This page collects some tips to aid the diagnosis of TLS-related issues and errors (TLS alerts). The strategy is to test the required components with an alternative TLS implementation in a process of elimination to identify the fault.

Bear in mind that this process is not guaranteed to identify the problem if the interaction between two specific components is responsible for the problem.

We also explain some of the most common error messages you may see in logs.

Check Effective Node Configuration

Setting up a RabbitMQ node with TLS involves modifying configuration. Before performing any other TLS troubleshooting steps it is important to verify config file location and effective configuration (whether the node has loaded it successfully). See Configuration guide for details.

Check TLS Support in Erlang

Another key requirement for establishing TLS connections to the broker is TLS support in the broker. Confirm that the Erlang VM has support for TLS by running the erl (or werl.exe on Windows) shell and entering

ssl:versions().
The output should look similar to this (with version number differences possible):
[{ssl_app,"5.3.6"},
 {supported,['tlsv1.2','tlsv1.1',tlsv1]},
 {available,['tlsv1.2','tlsv1.1',tlsv1]}]

If you receive an error instead, confirm that Erlang was built with OpenSSL. On Debian-based systems you may need to install the erlang-ssl package.

Check Keys and Certificates with OpenSSL

We will now verify the certificates and keys specified in the configuration file against an alternative TLS implementation. This example uses OpenSSL s_client and s_server. We will confirm that the certificates and keys can be used to establish a secure link by connecting two terminals. For the examples that follow, we will assume you have the following:

Item Location
CA certificate testca/cacert.pem
Server certificate server/cert.pem
Server key server/key.pem
Client certificate client/cert.pem
Client key client/key.pem

In one terminal window execute the following command:

openssl s_server -accept 8443 -cert server/cert.pem -key server/key.pem \
  -CAfile testca/cacert.pem
In another terminal window execute
openssl s_client -connect localhost:8443 -cert client/cert.pem -key client/key.pem \
  -CAfile testca/cacert.pem
If the certificates and keys have been correctly created, an TLS connection establishment sequence will appear and the terminals will be linked. Input from either terminal will appear on the other. If the trust chain could be established, the second terminal will display this confirmation:
Verify return code: 0 (ok)

If you receive an error, confirm that the certificates and keys were generated correctly and that a matching certificate/key pair is used. For environments where self-signed certificates are appropriate, we recommend using tls-gen for generation.

Check TLS Listeners (Ports)

This step checks that the broker is listening on the expected port(s), such as 5671 for AMQP 0-9-1 and 1.0, 8883 for MQTT, and so on.. When you start the broker with a valid TLS configuration file, the broker will report the TLS listening address in the logfile. You should see an entry similar to this:

=INFO REPORT==== 8-Aug-2011::11:51:47 ===
started SSL Listener on 0.0.0.0:5671
If you included an "ssl_listeners" configuration directive and you don't see this message, it is possible that your configuration file was not read by the broker or the node was not restarted after config file changes. See the configuration page for details on config file verification.

Attempt TLS Connection to Broker

Once you have a RabbitMQ broker listening on an TLS port you can again use the OpenSSL s_client to verify TLS connection establishment, this time against the broker. This check establishes whether the broker is likely to be configured correctly, without needing to configure an AMQPS client. The example assumes a broker with an "ssl_listeners" configuration directive set to listen for TLS connections on localhost port 5671:

openssl s_client -connect localhost:5671 -cert client/cert.pem -key client/key.pem \
  -CAfile testca/cacert.pem
The output should appear similar to the case where port 8443 was used. The broker logfile should contain a new entry when the connection is established:
=INFO REPORT==== 8-Aug-2011::11:55:13 ===
accepting AMQP connection <0.223.0> (127.0.0.1:58954 -> 127.0.0.1:5671)

It should now be possible to present the broker with an AMQP 0-9-1 connection header sequence over the TLS connection. If you present the broker with eight random bytes, the broker will respond with the string "AMQP" followed by an encoded version number. If you recognise the "AMQP" string, you can be confident that you are connected to a TLS-enabled node on the correct port.

Validate client connections with stunnel

The final check is to validate TLS-enabled clients. We will use stunnel to provide TLS capability. In this configuration clients will make a secure connection to stunnel, which will pass the decrypted data through to a "regular" port of the broker (say, 5672 for AMQP 0-9-1 and AMQP 1.0). This provides some confidence that the client TLS configuration is correct independently of the broker TLS configuration.

stunnel will run in daemon mode on the same host as the broker. In the discussion that follows it is assumed that stunnel will only be used temporarily. It is of course possible to use stunnel to provide TLS capability more permanently, but the lack of integration with the broker means that management reporting features and authentication plugins that use TLS information will not be able to do so.

In this example, stunnel will connect to the unencrypted port of the broker (5672) and accept TLS connections from TLS-capable clients on port 5679:

cat client/key.pem client/cert.pem > client/key-cert.pem
stunnel -r localhost:5672 -d 5679 -f -p client/key-cert.pem -D 7
stunnel requires a certificate and corresponding key. The generated client certificate and corresponding key should be used and concatenated as shown above with the cat command. stunnel requires that the key not be password-protected.

TLS-capable clients should now be able to connect to port 5679 and any TLS errors will appear on the console where stunnel was started.

Connect Client and Broker

If none of the previous steps produced errors then you can confidently connect the tested TLS-enabled client to the TLS-enabled port of the broker, making sure to stop any running OpenSSL or stunnel sessions first.

Certificate Chains and Verification Depth

When using a client certificate signed by an intermediate CA, it may be necessary to configure RabbitMQ server to use a higher verification depth. The depth is the maximum number of non-self-issued intermediate certificates that may follow the peer certificate in a valid certification path.

Refer to the TLS/SSL guide to learn how to configure verification depth.

Understanding TLS Connection Log Errors

New broker logfile entries will be generated during many of the preceding steps. These entries together with diagnostic output from commands on the console should help to identify the cause of TLS-related errors. What follows is a list of the most common error entries:

Entries containing {undef, [{crypto,hash,...
The crypto module is missing in the Erlang/OTP installation used or it is out of date. On Debian, Ubuntu, and other Debian-derived distributions it usually means that the erlang-ssl package was not installed.
Entries containing {ssl_upgrade_error, ekeyfile} or {ssl_upgrade_error, ecertfile}

This means the broker keyfile or certificate file is invalid. Confirm that the keyfile matches the certificate and that both are in PEM format. PEM format is a printable encoding with recognisable delimiters. The certificate will start and end with -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- respectively. The keyfile will likewise start and end with -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- respectively.

Entries containing {ssl_upgrade_failure, ... certify ...}

This error is related to client verification. The client is presenting an invalid certificate or no certificate. If the ssl_options has the verify option set to verify_peer then try using the value verify_none temporarily. Ensure that the client certificate has been generated correctly, and that the client is presenting the correct certificate.

Entries containing {ssl_upgrade_error, ...}

This is a generic error that could have many causes. Make sure you are using the recommended version of Erlang.

Entries containing {tls_alert,"bad record mac"}

The server has tried verifying integrity of a piece of data it received and the check failed. This can be due to problematic network equipment, unintentional socket sharing in the client (e.g. due to the use of fork(2)) or a bug in the client implementation of TLS.